A while back I announced on the PDO mailinglist that I would review the tests the mysqlnd team wrote. These tests are currently MySQL specific and I was hoping that we could adopt them for the other drivers. I spend less time than I originally envisioned on the review, mostly because the last 2 months were a bit more exhausting at work than I had anticipated. So I ended up sleeping more and working less. Anyways, at this point it seems to me like the tests themselves are too MySQL specific to be easily portable to other RDBMS and that the time is therefore better spend simply writing new tests and using the mysqlnd tests as inspiration. However, I feel like the current structure of the test suite is not optimal yet, though I might not fully grasp the current approach.

So I think the next step is to first address some of the short comings of the current approach or in case I just did not understand the current approach to make sure things are documented. The first issue I see with the current tests is that the core PDO tests are always just run against a single RDBMS. So if I just run the entire PHP test suite, I will only be testing one RDBMS. Of course I can easily write a script that changes the DSN environment variable and calls the PDO tests again, but I feel the default should be to run against as many RDBMS as configured. Alternatively every extension should include the core PDO tests when its called. That being said I think run-tests.php has no concept of "extensions" and instead tests are grouped by their location in the file system. Not sure if its easily nicely possible to have a "meta test" for each PDO driver extension that then sets the env variable and runs all the core PDO tests.

The much bigger issue I see is the fact that each RDBMS has differences that are not abstracted by PDO. I think the test results should make these differences visible or at least some of them. So syntax differences between RDBMS should not matter for the most part (except the parameter quoting, which PDO does handle) since PDO does not abstract these. This should mostly be trivial, but could get tricky in some cases. For example the syntax for LOB's might be different, but worse yet some RDBMS might require things to be inside a transaction, while others might not even support transactions and raise an error or warning.

However there should be a way to see differences in returned data, error codes, exceptions etc. So while the given behavior might be expected for the given RDBMS, it should be visible to people running the test suite that this behavior is a deviation from the "standard PDO" behavior. In this sense a test might need to be compared and reported against two different expected outputs. One time the PDO standard behavior and one time against the specific RDBMS behavior (in case it deviates from the standard behavior).

We could then write a script that takes this information and makes it available as a compatibility matrix. This would become an invaluable ressource for people that write applications or libraries that are actually supposed to work on top of different RDBMS. More importantly it would help in a mentally shift for driver authors. Incompatibilities would not get hidden by the fact that its the "expected behavior" for this driver. Or in other words incompatibility would be considered a bug. Of course not all incompatibilities are fixable, but they should remain known and transparent. Hopefully this will lead to a compatibility mode for PDO where some of the unnecessarily incompatibilities can be resolved without breaking BC (this actually makes me think that we also need to make it possible to run the test suite against different PDO attributes).

As such the PDO extensions should really only have specific tests for things that really do not work on _any_ other RDBMS. The second something works on at least another PDO driver it should be moved to the core tests. So some RDBMS have unique methods or queries and these should be the only specific tests. Any bug test case should ideally also be placed into core and not into the specific RDBMS driver that was affected.

So I guess the first project for this year is to figure out how to get the above done with phpt. After that it becomes time to actually sit down and write more tests. I am thinking that the first step should be completed by the end of this month and from then on I will try to write three tests per week. Hopefully others will join, but small steps are better than none. The tricky part in writing the tests will then be on deciding what the "PDO standard behavior" is and which driver gets to look non standard if there are differences. But thats another story ...

Lukas,

you can easily run PDO tests against all databases. run-tests.php has a "Redirect test" feature for this, just check ext/pdo_*/tests/common.phpt. That test checks the DBMS-specific DSN and then redirects to the general PDO tests using that DSN.

johannes