Recently on one project I tried to execute the Zend Unit Test Suite, but stumbled on a memory exhaustion problem. If I tried to set the PHP memory limit above 2GB, my computer crashed, probably when it tried to swap. I realized that if the 2GB memory on my desktop machine is insufficient, the 512MB on the cruisecontrol machine will never be enough for running the whole test suite.
On the CMAX.gg -project our coders implemented a custom shell-script based cruisecontrol system, that ran each SimpleTest test case through the web browser. On the current project I decided to use open source tools such as phpUnderControl and PHPUnit instead, so that I could use a readily available tool instead of trying to parse a new tool from the scratch again. Also, it seemed that the PHPUnit is more active and has nowadays more support than the SimpleTest.
After configuring phpUnderControl for several days, I realized that our old system was actually better than what is currently available. For example you need to restart Cruisecontrol.org Tomcat webapp each time you make changes to your config files, I needed to create separate scripts to check and restart Cruisecontrol when it crashes about once per week, and it uses a huge amount of memory and disk-space. I managed to reduce the disk-usage by disabling the phpDoc -target on each build. Yet more disappointments came when I realized it is not possible to run a single test case through the Cruisecontrol/phpUnderControl -interface and the graphs and the reports were almost unusable. If you are aware of the Agile practices, you really want to have only an overview, where you see the number of failed tests and the total number of tests. The phpUnderControl doesn’t sort the tests failed first unlike our own system does. I also realized already 5 years ago, that it is quite useless trying to draw a graph where you have the ratio or percentage of failed tests, since quite quickly you end up drawing a line with 99.9% and 99.8% pass ratios. A better graph is the absolute number of failed tests.
After Googling a bit, I ended also up to the PHPUnit Changelog, that advertised a new –process-isolation switch. The drawback was that I needed to upgrade from PHPUnit 3.3 to 3.4. After some experimenting I managed to run the tests, but they all failed to an unspecified error at an unspecified location like this:
124) Test_Ext_Zend_Zend::testSetState
RuntimeException: Warning: require_once(trunk/ext/Zend/tests/Zend/AllTests.php): failed to open stream: No such file or directory in trunk/test/Test/Ext/Zend/Zend.php on line 5
Call Stack:
0.0090 1206200 1. {main}() /home/pharazon/workspace/trunk/-:0
0.0093 1223456 2. require_once(’/home/pharazon/workspace/trunk/test/Test/Ext/Zend/Zend.php’) /home/pharazon/workspace/trunk/-:5
As you see, the stack trace says that the error happened at an file called ”-”, a quite hard trace to debug. After about a day of debugging I managed to get error messages to show by printing them out from the PHPUnit/Framework/TestCase.php ( print_r($job) ). After this I managed to figure out that PHPUnit creates dynamically a new PHP -file from a template for each TestCase. The basic Template is located at PHPUnit/Framework/Process/TestCaseMethod.tpl.dist . After more debugging, I managed to get most of the tests working by modifying the template and adding a bootstrap -file to setup the environment and the constants. PHPUnit –process-isolation uses reflection, var_export() and the magic __set_state() -method to serialize the TestCases on the fly. One hindrance was that the Zend -classes didn’t implement the __set_state() -method, so I was required to add them. The hardest ones to decipher were the Zend Translate -related classes, where just calling the constructor wasn’t sufficient. Finally I managed to create a function like this:
/**
* Magic reflection function to unserialize var_export’ed objects
* to be used with phpunit –process-isolation
* See more from http://www.thoughtlabs.com/2008/02/02/phps-mystical-__set_state-method/
* @param the same than for __construct()
* @author Pharazon
*/
public static function __set_state($data) {
return array_shift($data);
}
Compared to our own CMAX Cruisecontrol, it seems that the PHPUnit –process-isolation is a quite cumbersomely implemented and restricted. For example the current system doesn’t still enable running the tests on several cores, but only one thread is being used. Our Apache -based test runner was already 5 year ago able to run a multiple test cases per machine, and actually also distribute the running of tests across multiple test nodes. It’s quite easy through a http-interface, when Apache takes care of the multi-threading. Additionally the old system submitted the results on a test database. I think PHPUnit can do it also, but haven’t implemented it yet.
Next I started to package the solutions, but realized that using a custom statically modified PHPUnit and Zend was not probably the best idea in the world. Luckily I stumbled on the TestCase->prepareTemplate -function and realized it would be easy to create a subclass MyTestCase extends PHPUnit_Framework_TestCase that would implement the modifications without need for altering the PHPUnit source code statically. There are two changes required. The first problem is the loading of the bootstrap. The default template checks the loading of the file defined on the –bootstrap -command line parameter after importing of the globals. The problem was that the GLOBALS included the Zend -classes that were not yet introduced, since the bootstrap was not yet loaded. To solve this chicken-egg -problem the only solution was to create a custom TestCaseMethod.tpl that had a static bootstrap loader before introducing the GLOBALS. Additionally I added a new parameter called ’bootstrap’. Here is the new MyTestCase:
<?php
/**
* My Test Case that extends PHPUnit 3.4 TestCase, featuring
* – custom preparation to support bootstrapping with –process-isolation
* @author Pharazon
*/
class MyTestCase extends PHPUnit_Framework_TestCase {/**
* Performs custom preparations on the process isolation template.
*
* @param PHPUnit_Util_Template $template
* @since Method available since Release 3.4.0
* @author Pharazon
*/
protected function prepareTemplate(PHPUnit_Util_Template $template)
{
//use custom process-isolation template
$template->setFile(ROOT.’/test/MyCaseMethod.tpl’);
$template->setVar(array(’bootstrap’ => ROOT.’/test/testBootstrap.php’));
}
}
Then I changed all tests to extend MyTestCase instead of the PHPUnit_Framework_TestCase. Here is also the modified MyCaseMethod.tpl template:
<?php
set_include_path(’{include_path}’);//require_once ’{bootstrap}’;
require_once ’{filename}’;function __phpunit_run_isolated_test()
{
$result = new PHPUnit_Framework_TestResult;
$result->collectRawCodeCoverageInformation({collectCodeCoverageInformation});$test = new {className}(’{methodName}’, unserialize(’{data}’), ’{dataName}’);
$test->setDependencyInput(unserialize(’{dependencyInput}’));
$test->setInIsolation(TRUE);
$test->run($result);print serialize(
array(
’testResult’ => $test->getResult(),
’numAssertions’ => $test->getNumAssertions(),
’result’ => $result
)
);
}
{globals}__phpunit_run_isolated_test()
?>
I was happy when I got all the tests to pass :). But only for a moment…. Next I tried to get the tests to run on the CruiseControl / phpUnderControl -instance, but noticed that the PHPUnit 3.4 doesn’t support anymore PMD or the CodeCoverage -tools, but they have been extracted to separate processes. Doh! But it’s yet a new story that you will get later 🙂