Of Heaps and Queues

Queues in programming are any data structure that, when iterated, return values in a "first-in-first-out" (FIFO) order. For "last-in-first-out" (LIFO) iteration, you define a stack.

A heap is a data structure where, given a specific node, all nodes beneath it are of a value less than it. (Technically, this would be considered a "max-heap," as you can also have a variant where all child nodes are of a value greater; this is called a "min-heap.")

A priority queue is a specialized version of a max-heap. Typically, data is registered with a specific priority -- so the max-heap is looking at only the priority value, not the data itself. This allows inserting data into the queue in any order desired, while ensuring that they are iterated in the order specified by the priorities provided.

PHP offers SPL data structures corresponding to each:

• SplQueue, corresponding to a queue.
• SplStack, corresponding to a stack.
• SplHeap, corresponding to a heap.
• SplMaxHeap, correspondaxg to a max-heap.
• SplMinHeap, corresponding to a min-heap.
• SplPriorityQueue, corresponding to a priority queue.

Problems

The first problem I ran into was really a lapse of reasoning on my part, and is namely this:

Iterating over a heap removes the values from the heap.

Basically, in order to satisfy the heap contract, which is that the root node is always the maximum value (or minimum, in the case of a min-heap), any previous nodes must be removed.

The problem with this, obviously, is that if you want to iterate over a heap of any sort multiple times, well, you can't with the same instance.

The next problem I ran into was with SplPriorityQueue specifically: when items of equal priority are enqueued, the iteration order of these items is... unexpected. While the documentation notes that "multiple elements with the same priority will get dequeued in no particular order," the fact is that it is predictable, and unintuitive. For example, given the following:

$queue->insert('foo', 1000);
$queue->insert('bar', 1000);
$queue->insert('baz', 1000);
$queue->insert('bat', 1000);

foreach ($queue as $data) echo $data, \" \";

I'd expect a result of "foo bar baz bat", assuming FIFO order (which is expected in a queue) for equal priorities; "foo baz bat bar", assuming ordering by data (which might be expected in a max-heap). In fact, neither is true: the first item will be emitted first, and then the remaining items in reverse order of when enqueued: "foo bat baz bar".

While this may be somewhat predictable, I find I don't want to assume such order, nor try and write code around it.

Solutions

Allowing multiple iterations

Allowing multiple iterations of a queue is as easy as cloning it prior to iteration:

foreach (clone $queue as $datum) echo $datum, \" \";

The problem is automating this -- there are cases where I don't want users to really have to understand the internal implementation.

My solution to this was to use the idea of inner and outer iterators. In this particular case, I created a "PriorityQueue" class that composes an SplPriorityQueue instance, and which also implements IteratorAggregate. This allows the following:

namespace Foo;

class PriorityQueue implements Countable, IteratorAggregate
{
    protected $innerQueue;
    
    public function __construct()
    {
        // I'll explain the lack of global namespacing later...
        $this->innerQueue = new SplPriorityQueue;
    }

    public function count()
    {
        return count($this->innerQueue);
    }

    public function insert($datum, $priority)
    {
        $this->innerQueue->insert($datum, $priority);
    }
    
    public function getIterator()
    {
        return clone $this->innerQueue;
    }
}

This approach means that as I consume PriorityQueue, I can be assured that I can count and iterate over it... again and again.

I mention in the code comments that I'm not importing SplPriorityQueue into the namespace. The reason is that I want to also solve the problem of predictable queue order.

Enforcing predictable queue order

The solution to the queue order problem with equal priorities is actually quite simple. While I found it on the SplPriorityQueue::compare manual page, Matthew Turland also discusses it in a presentation on SPL, and it hinges on one, simple fact: priorities do not need to be integers.

What does this mean? It means that the following are not equivalent, and will lead to a more expected sort order:

$queue->insert('foo', array(1000, 1000));
$queue->insert('bar', array(1000, 100));
$queue->insert('baz', array(1000, 10));
$queue->insert('bat', array(1000, 1));

foreach ($queue as $data) echo $data, \" \";

This results in "foo bar baz bat"!

The trick, then, is automating the solution. I achieved this in a custom SplPriorityQueue extension:

namespace Foo;

class SplPriorityQueue extends \SplPriorityQueue
{
    protected $queueOrder = PHP_INT_MAX;

    public function insert($datum, $priority)
    {
        if (is_int($priority)) {
            $priority = array($priority, $this->queueOrder--);
        }
        parent::insert($datum, $priority);
    }
}

As each datum is added to the queue, if the priority is an integer, it wraps it in an array, using $queueOrder as a second value to the array, and decrementing $queueOrder on completion. The new priority is then used to insert the value.

Using this extension ensures that order in the priority queue is now predictable.

Conclusions

SplPriorityQueue is indeed powerful, and saves me a ton of time programming -- and also likely CPU processes and memory when using larger data sets. While it may not always meet my use cases, the fact is that, particularly with namespacing available, I can easily override the class to meet my needs.

My algorithm for generating the maps was absurdly simple:

• Scan the filesystem for PHP files
• If the file does not contain an interface, class, or abstract class, skip it.
• If it does, get its declared namespace and classname

The question was what implementation approach to use.

I'm well aware of RecursiveDirectoryIterator, and planned to use that. However, I also had heard of FilterIterator, and wondered if I could tie that in somehow. In the end, I could, but the solution was non-obvious.

What I Thought I'd Be Able To Do

FilterIterator is an abstract class. When extending it, you must define an accept() method.

class FooFilter extends FilterIterator
{
    public function accept()
    {
    }
}

In that method, you typically will inspect whatever is returned by $this->current(), and then return a boolean true or false, depending on whether you want to keep it or not.

class FooFilter extends FilterIterator
{
    public function accept()
    {
        $item = $this->current();

        if ($someCriteriaIsMet) {
            return true;
        }

        return false;
    }
}

I'll go into the mechanics of my criteria later; what's important now is knowing that a FilterIterator allows you to limit the results returned by your iterator.

I originally thought I'd be able to simply pass a DirectoryIterator or RecursiveDirectoryIterator to my filtering instance. This worked in the former case, as it's only one level deep. However, for the latter, it would only return the first directory level for all classes that matched -- i.e., if I ran it over "Zend/Controller", I'd get a match for each class under "Zend/Controller/Action/Helper/", but it would return simply "Zend/Controller/Action" as the match. This certainly wasn't useful.

I then discovered RecursiveFilterIterator, which looked like it would solve the recursion problem. However, I found one of two results occurred: either I'd receive an entire subtree if at least one item matched, or it would skip an entire subtree if the first item found failed the criteria. There was no middle ground.

The Solution

The solution was incredibly simple and elegant, once I stumbled upon it: pass my RecursiveIteratorIterator instance to the FilterIterator.

$rdi      = new RecursiveDirectoryIterator($somePath);
$rii      = new RecursiveIteratorIterator($rdi);
$filtered = new FooFilter($rii);

Really. It was that simple -- but, as noted, non-obvious. It also required a slight change within my filter -- instead of using current(), I'd need to first pull the "inner" iterator instance: $this->getInnerIterator()->current(). I show an example of that below when I go over the filter implementation.

As for my criteria, I had several options. I could require_once the file, and use the Reflection API to inspect the class to determine if it was an interface, abstract class, or class, as well as to determine the namespace. However, I couldn't be 100% sure the file would contain a class, so this seemed like overkill. That, and horribly non-performant, due to using reflection.

The next option was to simply slurp in the file contents into a variable, and use regular expressions. I love regular expressions, but in this case, it felt like I could possibly end up with some false positives. Also, since some of these files could be quite large, I was worried again about performance implications -- I don't want to have to wait forever to generate these maps.

The solution I went with was to use the tokenizer to inspect the file. Tokenizing is incredibly fast, and it's also incredibly simple to analyze the tokens.

I decided to store the detected namespace and classnames as public properties of the SplFileInfo objects returned; this makes it simple to iterate over the entire collection and utilize that information. Also, because I have SplFileInfo objects, I already have the paths I need.

My implementation looks like this:

/** @namespace */
namespace Zend\File;

// import SPL classes/interfaces into local scope
use DirectoryIterator,
    FilterIterator,
    RecursiveIterator,
    RecursiveDirectoryIterator,
    RecursiveIteratorIterator;

/**
 * Locate files containing PHP classes, interfaces, or abstracts
 * 
 * @package    Zend_File
 * @license    New BSD {@link http://framework.zend.com/license/new-bsd}
 */
class ClassFileLocater extends FilterIterator
{
    /**
     * Create an instance of the locater iterator
     * 
     * Expects either a directory, or a DirectoryIterator (or its recursive variant) 
     * instance.
     * 
     * @param  string|DirectoryIterator $dirOrIterator 
     * @return void
     */
    public function __construct($dirOrIterator = '.')
    {
        if (is_string($dirOrIterator)) {
            if (!is_dir($dirOrIterator)) {
                throw new InvalidArgumentException('Expected a valid directory name');
            }

            $dirOrIterator = new RecursiveDirectoryIterator($dirOrIterator);
        }
        if (!$dirOrIterator instanceof DirectoryIterator) {
            throw new InvalidArgumentException('Expected a DirectoryIterator');
        }

        if ($dirOrIterator instanceof RecursiveIterator) {
            $iterator = new RecursiveIteratorIterator($dirOrIterator);
        } else {
            $iterator = $dirOrIterator;
        }

        parent::__construct($iterator);
        $this->rewind();
    }

    /**
     * Filter for files containing PHP classes, interfaces, or abstracts
     * 
     * @return bool
     */
    public function accept()
    {
        $file = $this->getInnerIterator()->current();

        // If we somehow have something other than an SplFileInfo object, just 
        // return false
        if (!$file instanceof \SplFileInfo) {
            return false;
        }

        // If we have a directory, it's not a file, so return false
        if (!$file->isFile()) {
            return false;
        }

        // If not a PHP file, skip
        if ($file->getBasename('.php') == $file->getBasename()) {
            return false;
        }

        $contents = file_get_contents($file->getRealPath());
        $tokens   = token_get_all($contents);
        $count    = count($tokens);
        $i        = 0;
        while ($i < $count) {
            $token = $tokens[$i];

            if (!is_array($token)) {
                // single character token found; skip
                $i++;
                continue;
            }

            list($id, $content, $line) = $token;

            switch ($id) {
                case T_NAMESPACE:
                    // Namespace found; grab it for later
                    $namespace = '';
                    $done      = false;
                    do {
                        ++$i;
                        $token = $tokens[$i];
                        if (is_string($token)) {
                            if (';' === $token) {
                                $done = true;
                            }
                            continue;
                        }
                        list($type, $content, $line) = $token;
                        switch ($type) {
                            case T_STRING:
                            case T_NS_SEPARATOR:
                                $namespace .= $content;
                                break;
                        }
                    } while (!$done && $i < $count);

                    // Set the namespace of this file in the object
                    $file->namespace = $namespace;
                    break;
                case T_ABSTRACT:
                case T_CLASS:
                case T_INTERFACE:
                    // Abstract class, class, or interface found

                    // Get the classname
                    $class = '';
                    do {
                        ++$i;
                        $token = $tokens[$i];
                        if (is_string($token)) {
                            continue;
                        }
                        list($type, $content, $line) = $token;
                        switch ($type) {
                            case T_STRING:
                                $class = $content;
                                break;
                        }
                    } while (empty($class) && $i < $count);

                    // If a classname was found, set it in the object, and 
                    // return boolean true (found)
                    if (!empty($class)) {
                        $file->classname = $class;
                        return true;
                    }
                    break;
                default:
                    break;
            }
            ++$i;
        }

        // No class-type tokens found; return false
        return false;
    }
}

Note: the Exceptions thrown in this class are defined in the same namespace; I'll leave how they're implemented to your imagination.

Iterating Faster

The next trick I discovered was in the form of iterator_apply(). Normally when I use iterators, I use foreach, because, well, that's what you do. But in looking through the various iterators for this exercise, I stumbled across this gem.

Basically, you pass the iterator, a callback, and argument(s) you want passed to the callback. Like FilterIterator, you don't get the actual item returned by the iterator, so in most use cases, you pass the iterator itself:

iterator_apply($it, $callback, array($it));

You can then grab the current value and/or key from the iterator itself:

public function process(Iterator $it)
{
    $value = $it->current();
    $key   = $it->key();
    // ...
}

While you can use any valid PHP callback, I found the most interesting solution was to use a closure, as it allows you to define everything up front:

iterator_apply($it, function() use ($it) {
    $value = $it->current();
    $key   = $it->key();
    // ...
});

If you pass in a local value via a use statement, you can do some aggregation:

$map = new \stdClass;
iterator_apply($it, function() use ($it, $map) {
    $file = $it->current();
    $namespace = !empty($file->namespace) ? $file->namespace . '\\' : '';
    $classname = $namespace . $file->classname;
    $map->{$classname} = $file->getPathname();
});

Not only is this a nice, concise technique, it's also tremendously fast -- I was finding it was 200% - 300% faster than using a traditional foreach loop. Clearly it cannot be used in all situations, but if you can use it, you probably should.

So, start playing with FilterIterator and iterator_apply() if you haven't already -- the two offer tremendous possibilities and capabilities for your applications.