Poo-tee-weet

Seven things - tagged by a bunch of people

Lukas Kahwe Smith — Thu, 08 Jan 2009 19:24:52 +0100

I have been tagged by numerous people (Marcus, Lars, Mark and probably a few others .. just too man people playing along!).

So semi random facts about me ..

I actually picked my first name Lukas myself. At birth my name was Kahwe Haddad, but when I was adopted by the man I call my father in elementary school I changed my last name to his and also picked a new first name, moving Kahwe to be my middle name.
As some may have guessed from my birth name, I am half persian on my fathers side. My mother is from East Germany and actually escaped with her family from East Germany. My adoptive father is american. So I am a German, with a US Greencard, living in Switzerland. My step sister is American and my half brother is German.
So far I only weight over 60Kg once in my live, at the end of the week I spend in Quebec last year for the PHP Quebec conference 2008. My low weight is probably attributed to the fact that I am hypo glycemic (I build off sugar too fast). I am also dislexic, but that has no effect on my weight :)
Like so many others my first steps into programming in BASIC on some weirdo machine made by Apricot. Monochrome screen with an infrared keyboard no less!
Last year I played probably at 30 ultimate frisbee tournaments. Sometimes it feels like I am a full time frisbee player that occationally jobs as a programmer.
As a child/teenager I was mostly angry .. at everything .. like the lack of world peace or just the fact that someone did not hold the door for an old lady. At some point I just embraced being arrogant. Now a days I just shrug and say to myself "they don't know better, and since I do .. I have to help them". I also assume that nobody is evil and that people simply have a different perception of reality. I guess that helps dealing with internals@.
I am addicted to gadgets and software that is slick. One of the easiest way to appear slick in my eyes is to efficiently be top of the line at many things at once. So my winter jacket can be configured for optimal performance for any season of the year, sport activities including skiing etc.

So who am I tagging? I guess most people have been tagged by now:

Hali - because without you I would have committed suicide on my Oracle projects
Ulf - for the good work at MySQL AB
Edin - hoping this way I will hear something about you again
Dan - you really helped define a way for people working for big time companies to work within the PHP community
Dave - had a lot of fun working with you
Chregu - only because of Liip I could spend last year playing frisbee mostly
The PHP community - because you are all tagged to work together

Link your original tagger(s), and list these rules on your blog.

Share seven facts about yourself in the post � some random, some weird.
Tag seven people at the end of your post by leaving their names and the links to their blogs.
Let them know they�ve been tagged by leaving a comment on their blogs and/or Twitter.

PDO test suite

Lukas Kahwe Smith — Fri, 02 Jan 2009 15:31:58 +0100

A while back I announced on the PDO mailinglist that I would review the tests the mysqlnd team wrote. These tests are currently MySQL specific and I was hoping that we could adopt them for the other drivers. I spend less time than I originally envisioned on the review, mostly because the last 2 months were a bit more exhausting at work than I had anticipated. So I ended up sleeping more and working less. Anyways, at this point it seems to me like the tests themselves are too MySQL specific to be easily portable to other RDBMS and that the time is therefore better spend simply writing new tests and using the mysqlnd tests as inspiration. However, I feel like the current structure of the test suite is not optimal yet, though I might not fully grasp the current approach.

So I think the next step is to first address some of the short comings of the current approach or in case I just did not understand the current approach to make sure things are documented. The first issue I see with the current tests is that the core PDO tests are always just run against a single RDBMS. So if I just run the entire PHP test suite, I will only be testing one RDBMS. Of course I can easily write a script that changes the DSN environment variable and calls the PDO tests again, but I feel the default should be to run against as many RDBMS as configured. Alternatively every extension should include the core PDO tests when its called. That being said I think run-tests.php has no concept of "extensions" and instead tests are grouped by their location in the file system. Not sure if its easily nicely possible to have a "meta test" for each PDO driver extension that then sets the env variable and runs all the core PDO tests.

The much bigger issue I see is the fact that each RDBMS has differences that are not abstracted by PDO. I think the test results should make these differences visible or at least some of them. So syntax differences between RDBMS should not matter for the most part (except the parameter quoting, which PDO does handle) since PDO does not abstract these. This should mostly be trivial, but could get tricky in some cases. For example the syntax for LOB's might be different, but worse yet some RDBMS might require things to be inside a transaction, while others might not even support transactions and raise an error or warning.

However there should be a way to see differences in returned data, error codes, exceptions etc. So while the given behavior might be expected for the given RDBMS, it should be visible to people running the test suite that this behavior is a deviation from the "standard PDO" behavior. In this sense a test might need to be compared and reported against two different expected outputs. One time the PDO standard behavior and one time against the specific RDBMS behavior (in case it deviates from the standard behavior).

We could then write a script that takes this information and makes it available as a compatibility matrix. This would become an invaluable ressource for people that write applications or libraries that are actually supposed to work on top of different RDBMS. More importantly it would help in a mentally shift for driver authors. Incompatibilities would not get hidden by the fact that its the "expected behavior" for this driver. Or in other words incompatibility would be considered a bug. Of course not all incompatibilities are fixable, but they should remain known and transparent. Hopefully this will lead to a compatibility mode for PDO where some of the unnecessarily incompatibilities can be resolved without breaking BC (this actually makes me think that we also need to make it possible to run the test suite against different PDO attributes).

As such the PDO extensions should really only have specific tests for things that really do not work on _any_ other RDBMS. The second something works on at least another PDO driver it should be moved to the core tests. So some RDBMS have unique methods or queries and these should be the only specific tests. Any bug test case should ideally also be placed into core and not into the specific RDBMS driver that was affected.

So I guess the first project for this year is to figure out how to get the above done with phpt. After that it becomes time to actually sit down and write more tests. I am thinking that the first step should be completed by the end of this month and from then on I will try to write three tests per week. Hopefully others will join, but small steps are better than none. The tricky part in writing the tests will then be on deciding what the "PDO standard behavior" is and which driver gets to look non standard if there are differences. But thats another story ...

My advice to the database division at Sun

Lukas Kahwe Smith — Wed, 17 Dec 2008 21:55:59 +0100

More and more people are lending advice to the MySQL guys at Sun (not even mentioning the level of fact based slapping its getting along the way). I think there is one important thing missing in this advice, Sun has its feet in 3 RDBMS products: MySQL, Drizzle and PostgreSQL. Even if they paid a 1B USD for the first, the other two are important to remember as well. Actually Drizzle is mentioned a lot in this advice, but PostgreSQL is oddly absent. IMHO MySQL serves a market quite perfectly at this point and I do not see this changing in the next few years.

There are annoyances all around, but the fact of the matter is that we know them well and they do not stop us from succeeding with our web business. Sure proper FK's would be nice, sure non sucking subqueries would make some queries look nicer and yes I guess eventually it would be nice not to hand Oracle business by the way of InnoDB. But even so, isn't Sun a hardware company? So stay as friendly as possible with Oracle and things should continue to work fine (then again my points regarding PostgreSQL might change this). If you have to develop a new storage engine, then keep it at one at a time. Or better yet, scrap the dual licensing scheme on them. As long as you control the kernel what is there to worry about letting the community participate?

So just maintain MySQL, fix all the current features one by one focusing on what web developers need and not what super funkt data warehouse need. Essentially embrace what Percona and OurDelta are doing. They are focusing on adding the necessary instead of trying to please potential future target audiences. You are effectively giving the real world business to others in the hopes of one day getting to tastier budgets.

For that kind of stuff turn to PostgreSQL. Its there, it rocks for this kind of stuff. It fits much better into your hardware business and old school Oracle/DB2 DBA's will have a much easier time to get their heads around it than MySQL. Do not try to make MySQL into something it was never designed to be (intentional or not). Make sure that PostgreSQL continues to learn to scale with as many cores as you give it. Make sure that you leverage all that networking you have in place via your MySQL developers to get the community to understand where they should try to see PostgreSQL as the platform to support.

Now Drizzle. I do not get cloud computing (maybe someone can explain it to me). Its fancy, its hype and I am sure there are plenty of people that need it, next to the other 99% that do not. Maybe this cloud computing thing was just a way to sell the thing to the Sun sales department. Dunno. Most of us today start of with a db installed along with a web server on the same machine. When we need to scale we eventually reluctantly setup a master, but keep a local slave on our web servers. This is todays boring reality. Now if you guys are 100% that in 5 years time we will all do is cloud computing, then make sure that in 2-3 years time we already start using Drizzle for something or chances are that someone else comes along around that timeframe that will do just that and have an installed user base for when cloud computing will really start to make a dent. Either way, plan Drizzle to become the successor of MySQL. Plan to make it a better solution to the needs of web developers in 5 years time than MySQL or other competitors will be in 5 years time.

Oh and on a finally note. Ignoring windows is a bad bad idea and unnecessary (referring to Jay's post here). Every year more and more open source stuff makes its way to native windows versions, so for most (everything?) there are sensible cross platform solutions around. But accept the fact that most web developers are in on windows. Maybe not in 5 years time (ok probably still even then), but definitely in 2-3 years time (the timeframe when you want to build your user base to be able to rule in 5 years).

Ah and a final final note. Forget about this enterprise-community edition confusion stuff. Just focus on being the goto company for support when its time to setup an OSS based database setup (or any database setup for that matter) and have your hardware be the natural combo.

Anyone played with SQL Relay?

Lukas Kahwe Smith — Sun, 14 Dec 2008 21:10:20 +0100

Since I recently stumbled over issues with persistent connections I got reminded once again about SQL Relay. Not really a solution to my problems, since SQL Relay will not magically clean up open transactions (or does it?). However it would help reduce the number of open connections while still getting rid of most the connection overhead. Then again at peak times it might not reduce the number of connections all that much and at low traffic times I guess the main benefit is that one can have more spare children hanging around without clogging up the DB with lots of open connections.

Still this makes me wonder if its even worth it, as I am not sure how much it will really safe me at high load times, as at least in my current app it seems impossible to get even a single request that does not have to hit the DB. So maybe SQL Relay is more useful for apps where not every request needs a DB connection or where the DB connection is not needed for a larger chunk of the request and so the connection can be returned to the SQL Relay pool.

More annoyingly it seems that there is no PDO driver yet, but just an odd API "native" extension. Worse yet it all seems to be GPL'ed, which makes it impossible to get into PECL and as a result it will probably never get out of its niche in the PHP community. So anyone played with it and give me some feedback on of its worth to dive in or if you agree that an application that needs to talk to the database for the most part while handling a request is not going to benefit?

On scaring your audience shitless

Lukas Kahwe Smith — Sun, 14 Dec 2008 13:02:53 +0100

Well it really wasn't my intention, but I was aware of the risk when I prepared this talk at Forum PHP in Paris. I held the closing session where I talked about what I have learned while trying to document and eventually becoming part of the "PHP development process". The gist is essentially that there is not much of a formal process or any single person/entity that is controlling things. It all works based on trust, mostly implied rules/guidelines .. although increasingly the implied stuff is getting documented. Sounds scary? Well it sounds quite reassuring to some and scary to others.

Reassuring to those that do not trust a single person/entity. I guess among the open source crowd this is actually the majority of people. But scary to those who are not yet to deep into open source. This is probably the majority of PHP developers as their choice towards PHP was likely driven by other aspects than open source. Ease of use, scalability/performance and price, the later of which is obviously a result of the fact that PHP is open source, but for all most people could care for it could be public domain.

So given this, I probably tried to speak a bit more towards the part of the audience that is scared off by trust networks. I should have mentioned that PHP is the #1 platform for the web. That it has been successful for many years already and looks back at a history of over 10 years (which is 100 years in Internet years). That the fact that most of the PHP Groups does other stuff these days and that a fair number started disappearing before PHP was really big, means that PHP is healthy enough that new developers keep coming in. Every year we welcome at least a dozen (this is just a guess as I did not try to get the actual numbers) core developers and here I am counting those who really start to become part of the active core developers. There are probably 2-3 dozen of new CVS accounts opened for php-src each year. But when it all comes down to it, people need to accept that PHP's success is a testament that humans can work together in a network solely based on trust to build something good for all.

As a means to bring more people into this trust network I once proposed the emPHPower idea. But I got kind of stuck with too little time ever since I started co-RMing PHP 5.3. Also nobody really jumped on the idea in the sense that they really pushed me to do something. Now things have flared up once again while talking to people at Forum PHP. Also after hearing David's talk on Ning I decided its better to take the risk of launching emPHPower on top of a closed source platform, than to do nothing. David reassured me that if we ever do decided to move, we could get a dump (fairly raw of course) of all the data. So lets see where this takes us ..