Poo-tee-weet

Oracle needs to get hurt .. badly!

Lukas Kahwe Smith — Fri, 13 Aug 2010 21:59:51 +0200

There are plenty of bad things in the world. But I guess there is nothing that has such a drastic and dangerous impact on my life that software patents. If patents are good thing in other industries can be discussed, but in the software industry its a clear cut thing: they hurt innovation, they hurt small businesses and they scare the shit out of me. And if you are a software developer, they should scare the shit out of you too! Now Oracle decided as the first big company that is not just a patent troll to actually sue a company over software patents. heck maybe others have sued before, but lets stop this behavior right here right now. We must send a clear message. We must send a clear message to Oracle that they better stop. And we must send a clear message to any other company holding software patents that they better not think of suing. I have not thought this out all the way yet, but right now I am thinking the best form of protest would be if all of us open source developers strip out support for any Oracle product in our next release: Throw out support for BerkleyDB, MySQL, Oracle whatever. The business math must be clear: Suing over software patents will cost more then it can ever get in returns. Who is with me in organizing such a protest? Or does someone have a better idea of how to approach this?

Maybe I should clarify: The point is not to hurt any of the open source projects that would be affected by such a move. Stripping out the support for one release would of course be a nuisance for users, but it would not be a big issue in the long run. Of course if Oracle doesn't react we might need to continue or to turn up the heat. It might actually lead to a fork or two if Oracle decides to keep this up. In this case I am also hoping that this will cause a few OSS guys at Oracle to hand in their notices .. even more people willing and capable to lead forks. Anyway the stronger our immediate message the lower the chances we need to be doing anything for any extended period of time.

Red tape and the art of ripping through it

Lukas Kahwe Smith — Sun, 18 Jul 2010 18:06:40 +0200

It seems like the Symfony2 CS is forbidding the use of private methods. I think other frameworks have similar rules. Now I have argued along those lines myself in the past. But especially talking to the Doctrine2 dev's got me to reconsider .. a bit. This all kinda of reminds me about the discussion over making it a fatal error to change method signatures during inheritance. I love PHP for allowing me to do this, yet giving me an E_STRICT, aka red tape, to make sure I am aware of the kittens that are being strung up this very moment. When I have coded myself into a corner, I need to get out of it quick, I do not need

I still do not believe in private, but I do believe that it would be a good idea to keep the user in the loop what the extension points are that you designed and where you commit to maintaining BC and where you don't. In many cases users may want to change a private method simply because they didn't understand the design of the API (often because they are too lazy to read the docs to learn how to do things right). Without some indication which methods are public, protected and private users would not know that they are over stepping red tape and get upset if methods are changed later on. So again I think it would be awesome to just get a slap on the wrist, aka an E_STRICT, when violating visibility rules. Since this is not the case, I actually kind of like the CakePHP approach using single/double underscore's for visibility constraints.

Now I should explain a few more things about the benefits doing the right thing� and the reasons for still sometimes not doing the right thing�. In general if an API is well designed use of private can make total sense. Basically you are saying this part of the code is not extensible, it hasn't been designed as such. Such a great designed API should however have more sensible extension points to still get the results you want. Similarly, breaking the "is a" relationship by screwing with the signature of methods will eventually bite you in the ass too, because suddenly assumptions about what you can do with an object because it is an "instance of" something will blow up in your face. Doing the right thing is the right thing most of the time.

However the reality of project life is that some times you need to shot from the hip to get things done in time and on budget. Sometimes this might code you into a corner, but it will still be good enough for the project to be a success. Again this is why I appreciate the red tape, that way I cannot accidentally do the wrong thing�, it will have to be a conscious decision and one that I cannot easily hide from the client (if they have any knowledge of PHP). I do not want to hide anything from the client, I want them to be aware of the compromises I make. But if their time and budget does not allow for a perfectly thought out design or when a time saving library simply doesn't allow me to do the things I need to do for a client, then I want to be able to overstep the red tape, because it is the best thing to do for my client. There might be clients that rather want to spend the time, and I very much welcome these kinds of clients. But I fear most of these clients rather go to a Java shop for that kind of work and they are willing to accept the additional required development time and budget requirements.

I also want to present another angle. The internet thrives on the idea of innovation at the edges. People put out things and others pick them up, twist and turn them until they do things never imagined possible by the original people. More importantly things move way faster than the original people ever anticipated and chances are good that they will have a hard time keeping up with their users. Having to wait for some upstream project to integrate some patch, or even just agreeing that a change is necessary is often just not possible in the context of a client project. Of course DVCS like git have made all of this more possible: You simply fork on the spot, easily keep up with upstream changes and when things slow down, you can contribute the changes back. But what happens if they never accept your changes or add the feature in an incompatible way? Then you are stuck with maintaining your fork, which will become am increasing pain if they make BC breaking changes upstream, just like if you might see BC breaks if you rely on extending private methods (assuming the language would allow it). So I much prefer if I could have some nagging E_STRICT remind me of the places where I over stepped the red tape.

Now knowing when it is ok to overstep the red tape, knowing the chances of this biting you in the ass, knowing where to look when things are biting you in the ass etc.: These are all things that set apart the good from the great PHP developers. This is the main difference between a junior and a senior developer and they have to take responsibility when things go wrong. Anyway, I guess the point with use of private is that you really need to know what you are doing to end users and you better spend enough time designing a very good API because you basically have taken away the ability of the end user to fix your oversights.

Transforming end user queries to Solr

Lukas Kahwe Smith — Thu, 01 Jul 2010 10:06:56 +0200

A bit less than a year ago I last did a presentation about a telephone book application where we used SQL to do some fairly advanced filtering over about 30 tables of data. The app generated SQL statements that filled pages, the more terms the more pages, but on a 10k dataset it still came back within a few milliseconds, thanks to a ton of indexing and denormalization tricks (SQL Server is a lot more powerful here than MySQL) I had applied. Now in a more recent project I am dealing with 10M+ dataset running on MySQL and so decided to learn about Solr. Wow, that thing is amazing and way more flexible in terms of query language than I expected. As a result I do not see it any more for just projects that are too big for an RDBMS, but more as the way to do search in general. I have mentioned resolutionfinder.org a few times (used to be called UN-informed.org). Solr is a key piece there and more importantly I am looking to expand the use of Solr query language quite a bit. Actually for those who know, you can already do a lot more powerful queries, something Liip will be investing some more time to make more accessible to end users with some UI tweaks planned in July. But in this blog post I want to talk about a prototype class I threw together (Look ma', I'm using git!) by working ezcSearch to help me in parsing and transforming end user queries into complex Solr queries.

Right now we have 3 input boxes on the resolutionfinder.org start page. The first one lets user input terms which are searched in the document title and the clause content. Here I recently exposed some of the syntax supported by the dismax handler like "+" to require and "-" to prohibit terms. Next is an input box that doesn't directly affect the search filtering. What it does is use auto suggest to build up a list of tag to filter on. The third input box actually is a separate search form entirely which is used for searching for specific documents by document code. My vision is to bring these all together into one input box. So if someone wants to do a fulltext search they could do something like the following:

"security council" +africa tag:malaria code:A/RES/*

This query would be parsed and result in looking for documents that must contain "africa" in the document title or clause content (scoring those that also contain "security council" higher), which are tagged with "malaria" and who's document code started with "A/RES/". This requires quite a bit of transformation. Actually the final queries I want to construct looks like this:

q=tag_ids:23 AND document_code_prefix:(A/RES/) AND (_query_:"{!dismax qf='content document_title' pf='content document_title' mm=0 v=$qq}")&qq="security council" +africa

So what happened here? As you can see I transformed "tag" to "tag_ids" and "code" to "document_code_prefix". These are the internal fields I use in the Solr index. There is actually also a "document_code" field, which I would use if the code would not end with a "*". The "document_code_prefix" just copies the "document_code" and applies an ngram filter to make prefix searches super fast. Also you may notice that instead of "malaria" I am filtering for 23, this is because right now I just index the tag id's and not the names. The next bit is taking all the un-fielded filters and apply a dismax search handler on them. In order to not require fancy escaping I use "v=$qq" to define that the actual search terms are passed in the "qq" GET parameter.

With the above mentioned class this sort of thing becomes fairly simple. I am using the symfony sfSolrPlugin, which provides query construction tools via the sfLuceneCriteria object. All I have to do is use the parser to parse the end user query into tokens and then use a custom term class that uses the sfLuceneCriteria instance to handle the serialization.


<?php
$criteria = new sfLuceneCriteria;
$stack = array(new phpSolrQueryTermCustom('AND', $criteria));
$parser = new phpSolrQueryParser($factory);

$tokens = $parser->parse($q, $stack);
$terms = $parser->processTerms($tokens);
var_dump($q);
var_dump($terms->serialize());
?>

You can check out the examples to get some more details about what is possible. Following is the code to the example on github, but here is a link to the actual term class and action class used in production.


<?php
class phpSolrQueryTermCustom extends phpSolrQueryTerm {
    protected $criteria;

    public function __construct($op = '', $criteria = null)
    {
        $this->op = $op;
        $this->criteria = $criteria;
    }

    protected function processTerm($term) {
        $prefix = $term->prefix;
        $field = $term->field;
        $term = (string)$term;
        $op = $this->op;
        $op = rtrim(" $op ").' ';
        if (!empty($field)) {
            switch ($field) {
            case 'code':
                $field = substr($term, -1, 1) === '*' ? 'document_code_prefix' : 'document_code';
                $term = $field === 'document_code_prefix' ? substr($term, 0, -1) : $term;
                break;
            case 'tag':
                $field = 'tag_ids';
/*
                $q = Doctrine_Query::create()
                    ->select('t.id')
                    ->from('Tag t')
                    ->where('t.name = ?', array($term));
                $term = $q->execute(array(), Doctrine_Core::HYDRATE_SINGLE_SCALAR);
*/
                $term = rand(1, 100);
                break;
            default:
                throw new Exception('Unsupported field: '.$field);
            }
            $this->criteria->addField($prefix.$field, $term, $op, true);
            return;
        }
        $term = parent::processTerm($term);
        return $prefix.$term;
    }

    public function serialize()
    {
        $terms = $this->terms;
        $op = $this->op;
        $op = rtrim(" $op ").' ';
        $dismax = array();
        foreach ($terms as $key => $term) {
            if ($term instanceOf phpSolrQueryTerm) {
                $term = $this->processTerm($term);
                if (!is_null($term)) {
                    $dismax[] = $term;
                }
            } else {
                $dismax[] = parent::processTerm($term);
            }
        }
        if (!empty($dismax)) {
            $dismax = implode(' ', $dismax);
            $subcritieria = new sfLuceneCriteria;
            $subcritieria->add('_query_:"{!dismax qf=\'content document_title\' pf=\'content document_title\' mm=0 v=$qq}"', $op, true);
            $this->criteria->add($subcritieria, 'AND');
            $this->criteria->addParam('qq', $dismax);
        }

        $q = $this->criteria->getParams();
        $q['q'] = array($this->criteria->getQuery());
        return $q;
    }
}
?>

Deploying app updates to a cluster

Lukas Kahwe Smith — Mon, 21 Jun 2010 11:23:01 +0200

So William was asking on twitter how to best deploy symfony apps to a cluster of servers. There are actually some nice deployment tools inside the symfony cli that ease deployment to a single server, but that doesn't really cover the cluster case. Actually I assume that if you have a cluster of servers the best deployment strategy should probably be optimized against your specific use case. But let's make this question a bit more general: How do you deploy updates to your PHP apps running a clustered setup? What architecture do you pick? How do you keep the site running with as few limitations as possible during the update? How do you distribute the new code? How do you clean and prime your caches? How do you handle DB changes? How do you ensure that the DB and code changes do not get in the way of each other?

Obviously the choice of RDBMS can play a big role here. MySQL likes replication setups, so I can envision making the site read only, disabling replication, update master schema, take a slave out of the load balancer, update code, reenable replication to that slave, repeat. With PostgreSQL, Oracle and RDBMS like that you tend to scale with just getting a bigger DB server (and having a warm failover server). Guess that simplifies things a bit (especially since PostgreSQL even supports transactional DDL, which should help with timing the deployment of the DB changes). Or maybe just make sure that DB changes never cause BC breaks.

Anyway, I am looking to just collect approaches here in the comments. So please ideally post links to slides or detailed blog postings below. I am sure this could become a very useful link collection for the PHP community. Thanks :)

Vendors: please stomp out SQL injection

Lukas Kahwe Smith — Sat, 12 Jun 2010 17:06:23 +0200

I have blogged about prepared statements a few times, which is what most people rely on (too much) for SQL injection protection. I say too much because they do not really protect code fully against SQL injection attacks and they come with a lot of performance hurting baggage. To sum up: prepared statements do not handle all aspects of dynamic SQL creation, they add network I/O and memory overhead and they tend to generate less optimal query plans. Some of these issues can be solved by doing client side emulation, but that brings with itself its share of issues and I have to agree with Bill and not Brian that parsing SQL should be left to the server.. So vendors, how about it? How about offering us a proper solution to prevent SQL injection attacks like I was asking for in my last post on the topic?

Again here are the requirements:

a single round trip for a single execution (prepare and execute with one command)
no overhead for single execution (option to not return a handle for additional executions, option to optimize the query plan for the first set of values)
handle scalar placeholders, as well as lists of scalar values (so also handle stuff like IN)
handle identifiers and operators (so for example to handle dynamic ORDER BY)
easy debugging (some way to return the final generated query)

Doesn't seem too hard to implement. I like how Oracle does a lot of things here and maybe they will teach some of that to MySQL now that its "their baby". I like the ":name" syntax for the placeholders, but more importantly the fact that they generate the query plan for the first set of values. However recently I found myself using the question mark syntax more and more, even though PDO can switch between the two versions seamlessly (well the parser has its share of issues). Maybe for other types (identifiers and placeholders) another syntax could be used like (exclamation mark maybe?) to ensure that one does not accidentally enable more flexibility in a parameter than intended.