Category Archives: Testing

Y2038 and Sustainable Software

Coding Testing Work Psychology

Today we were fixing some old unit tests for our large marketplace software, when I stumbled upon a peculiar test case. The test, written in 2005 assumed that a suitable date far in the Future would be 2015-01-01, and was now failing.

We fixed the test case by the DoTheSimplestThingThatCouldPossiblyWork and extended the future date until Y2038.

Anyway, the morale of the story is that at least at PHZ.fi, where we endeavor to build Sustainable Software, let’s expect that our software will be used for more than 10 years, before somebody rewrites it! In our case we are still using more than 10 year old software, but since we have unit tests (that now pass 100% again), it can still produce (a lot) business value. For a developer building sustainable software over building low quality, throw-away software, makes a big difference on the impact you contribute to the world.

 

Published by:

New company tagline – In Tests we Trust

Company Testing

Today we launched a new company slogan

In Tests we Trust

to promote our mission to provide sustainable software that can be maintained and developed further indefinitely. If your application is well covered by automated tests, you can easily refactor, add new features, debug and add more business value without having ever the need to start over and rewrite the software from a scratch. This leads in to huge cost savings over the application life-cycle.

Often, in many projects, we have seen that the typical half-life of an application is roughly 18 months, i.e. the time it takes before someone starts to talk about creating the version 2.0, meaning a complete rewrite of the code base. This is only a symptom of a bad software development process, and in particular lack of test automation. It means that the cost to make a change has grown greater than starting your project over. Multiply the monthly salaries of your coders by 18 months to get the cost. It is a great waste to throw away your investments every second year.

However, in PHZ.fi, we believe in test automation and sustainable software development. We have been successfully maintaining complex applications and legacy code for a decade, without having ever need to consider starting over from a scratch. Over time we have calculated over -95% application life-time cost savings compared to more traditional projects.

When our automated test cases show green, we can get a good night’s sleep.

Published by:

How to test file download by Selenium

Testing

In principle it’s a bit tricky to test file download by using PHPUnit and Selenium, because you can’t control the browser windows (such as the file download dialog) by Javascript. There have been other approaches to overcome this limitation, such as using AutoIT or AWT Robot/JInvoke (but it restricts your tests to Windows only or installing some new tools). This generic approach works in other languages such as Java, Ruby and Python, too.

This approach has also some limitations, it works only in Firefox (in theory it can be used in any browser that can be configured to auto-download to a specific directory without popping up the file download dialog). This requires also some detailed configuration and modifications to the both selenium-server and browser settings, but you don’t need to install any extra libraries or tools. I figured out this test case when I was required to automatically test downloading of a CSV Excel file (or in particular test that an empty file is not downloaded).

1) Create a Custom Firefox Profile

I found this instruction from here (you might need some alternations if you use for example Seamonkey or Firefox 2), but this worked at least on Firefox 7.0.1. First create a directory in your project root called for example Test/BrowserProfiles/firefox3.selenium/ and create a file called mimeTypes.rdf with the following contents:

<?xml version="1.0"?>
<RDF:RDF xmlns:NC="http://home.netscape.com/NC-rdf#"
         xmlns:RDF="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <RDF:Description RDF:about="urn:root"
                   NC:en-US_defaultHandlersVersion="3" />
  <RDF:Description RDF:about="urn:mimetype:application/vnd.ms-excel"
                   NC:value="application/vnd.ms-excel"
                   NC:editable="true"
                   NC:description="CSV document"
                   NC:alwaysAsk="false"
                   NC:saveToDisk="true">
    <NC:fileExtensions>csv</NC:fileExtensions>
    <NC:fileExtensions>xls</NC:fileExtensions>
    <NC:fileExtensions>xlb</NC:fileExtensions>
    <NC:fileExtensions>xlt</NC:fileExtensions>
    <NC:handlerProp RDF:resource="urn:mimetype:handler:application/vnd.ms-excel"/>
  </RDF:Description>
  <RDF:Description RDF:about="urn:mimetype:handler:text/csv"
                   NC:saveToDisk="true"
                   NC:alwaysAsk="false" />
  <RDF:Description RDF:about="urn:mimetype:text/csv"
                   NC:fileExtensions="csv"
                   NC:description="CSV document"
                   NC:value="text/csv"
                   NC:editable="true"
                   NC:alwaysAsk="false"
                   NC:saveToDisk="true">
      <NC:handlerProp RDF:resource="urn:mimetype:handler:text/csv"/>
  </RDF:Description>
</RDF:RDF>

The key thing here is to set the two properties NC:alwaysAsk=”false” and NC:savetoDisk=”true” to get the file downloads to start automatically without popping the file download dialog. You can also change and set the NC:fileExtensions and mime types (NC:Value) to other than text/csv, too, to suit your needs.

The original approach was to copy your current whole Firefox profile directory and then edit the files, but this incorporated hundreds of unnecessary files to the project. Since we have a quite strict code review process and nobody wanted to line-by-line inspect binary cache files, I figured out that you don’t actually need any other files than the mimeTypes.rdf, and that you can delete most of the contents of that file without any harm.

2) Customize Selenium-Server to use the custom profile

Start the selenium-server by the following command line:

java -jar selenium-server-standalone-2.6.0.jar -firefoxProfileTemplate ~/workspace/myproject/test/selenium/browserProfiles/firefox3.selenium

Please edit the paths and jar versions according to your own setup.

3) Write the test case

This is a bit simplified version of the real test case I made, for clarity. The approach is to first check from the (Firefox) Downloads/ folder that the file does not exist there yet. Then the download-button is clicked, and assuming everything is set up as described above, Firefox should download the file without asking any questions. Then you can check again the Downloads/ folder for the filename and check if it has appeared.

<?php
require_once('PHPUnit_Extensions_SeleniumTestCase');
class DownloadCSVTest extends PHPUnit_Extensions_SeleniumTestCase
{
    /**
     * Test Case - Try load a CSV
     * Expected: CSV file was downloaded
     * Note! This works only in Firefox and if you use the custom browser profile
     * in Test/Selenium/BrowserProfiles/firefox.selenium to download the file
     * This test assumes that your Firefox download directory is ~/Downloads/ (Linux)
     * Note! You need to launch your selenium-server to use the firefox profile found in
     * java -jar selenium-server-standalone-2.6.0.jar -firefoxProfileTemplate ~/workspace/ServiceProductionAdmin/Test/Selenium/BrowserProfiles/  firefox.selenium/
     * @see http://kb.mozillazine.org/File%5Ftypes%5Fand%5Fdownload%5Factions
     * @author Antti Hätinen <antti.hatinen@phz.fi>
     */
    public function testCSVDownload()
    {
	//this works only in Firefox, so use browser type *chrome
        self::$browser = "*chrome";
        //clear up
        $filename = getenv("HOME") . '/Downloads/fileToBeDownloaded.csv';
        @unlink($filename);

        //Download
        $this->focusAndClick("button:contains('Download')");
        sleep(3); //wait for download to complete
        $this->assertTrue(file_exists($filename),"CSV was not downloaded to '$filename' . Check that you have configured the Selenium-Server to use the custom firefox profile in myproject/Test/Selenium/BrowserProfiles/firefox.selenium/");

        //delete the file if it exists
        unlink($filename);
    }
}

Then you can run the test

 phpunit --stop-on-failure --filter testCSVDownload DownloadCSVTest.php

And hopefully all tests pass (sooner or later :).

Published by:

It’s Not Your Fault – Blame the Designer

Software Engineering Start-up Testing

Most people don’t realize how often they actually do mistakes. Every wrong move or thoughtless action, even if you can recover it immediately, is still a mistake that could have been possible to perform flawlessly. In general, people don’t admit doing errors, especially publicly in fear of becoming shamed. Six Sigma suggests as the source of the most of the cost in the products the so called Hidden Factories, or error corrections that are performed by the employees without informing the management, to avoid taking the blame, but incurring nevertheless the cost of the error. However, the belief that the mistake that you just made would be your fault is actually not true! If you just made an error, it’s actually not your fault! Instead, you should point your finger to the designer.

At Toyota the Kaizen continuous improvement and learning organization uses the following framework for blaming and learning from mistakes. You can adapt this philosophy also on your personal life, but I discuss it from the entrepreneur’s point of view.

1. Blame the Instructions

If you don’t have instructions how to perform a specific task, the work design and management is to be blamed. The definition of the management and entrepreneurs is actually not do any work, but to manage the organization of work, meaning how exactly you should perform your tasks and which tasks should be performed in the first place. The work of an entrepreneur is to organize the factors of production so that you can convert ideas into practice filling out a market niche. The work of the managers is to find optimal organization of the work for a given problem. The worker should not be interested of the management of the work, since it’s not his primary interest, as long as the company stays afloat and employment is guaranteed. Similarly, a manager or entrepreneur who is not interested in the continuous improvement of the work processes is not performing his job properly.

The current and western management nomenclature tends to omit the first and primary factor of production as irrelevant – the standardization of work. However, the as any basic economics book can tell you straight-on, without standardization of work there is no economies of scale and the work remains ad hoc in nature and economically unfeasible. In the western countries the emphasis is put on the innovation, or the Deming’s PDCA -cycle, how to improve the current work process by radical innovation. Unfortunately most often the basics are forgotten in the fuzz of innovation and the management fails to standardize the work after or before the innovation, resulting in utter chaos and high costs. No wonder that the jobs are continuously being outsourced to China and India.

Secondly, if by a miracle a standardized work process would exist in your organization, the instructions might well be incomplete, inaccurate and ignorant of the most common error modes. The first corrective action at Toyota is aimed thus on improving the instructions so that the by following them the work can be performed succesfully.

Yet a more advanced philosophy for standardization is the Six Sigma -approach that tries to identify and mitigate the sole possibility of producing an error. The Japanese call this principle as Poka-Yoke, or error-proofing. The idea is that by eliminating the source cause for errors the work process can be performed flawlessly producing significant quality and cost improvements.

2. Blame the Machines

If despite improving the instructions you still make them, you must turn the look on the machinery and tools that you are using to perform the task. For example if your computer is crashing and preventing you from performing your work, it’s not the fault of the instructions or yourself, but of the computer. Thus you should find a new tool or machine design that makes it impossible to perform such an error.

The Designed for Six Sigma (DFSS) -method tries to achieve exactly this. 80% of the quality problems can be tracked back to the design board, and thus it is not possible to reach the higher levels of quality (5 and 6 sigma) without taking the high quality in account already on the drawing board. The DFSS tries to eliminate the potential for producing Critical-to-Quality (CTQ) error in advance by for example preferring usage of proven components, analysing the potential error modes (FMEA) and the Voice of Customer.

The Japanese have developed a 14 level model of intelligent machine automation (Jidoka). The idea is to convert the machines to automatically detect errors in themselves. The idea comes from Toyota’s founder’s Sakichi Toyoda’s automatic weaving machine that automatically detected a broken loom. Six Sigma suggests of using simple automatic pass/fail -gates for investigating the quality level on a production line. The same idea is called as Test-Driven Development in software engineering, where you before starting of the coding define the end product by writing automated unit and acceptance tests.

3. Blame Yourself

If you are not able to follow the instructions or the standardized process, it’s your fault. In this case Toyota considers a task for you that you are able to perform according to the instructions. However, before blaming on the human factors, one should consider the two previous steps if they have rendered the work inhumane to perform, and if the blame can be still be placed on the Work and Machine Design first.

Of course, if you are unable to follow the Kaizen continuous improvement -cycle, the blame is truly only on yourself :)

Published by:

PHPUnit 3.4, –process-isolation and phpUnderControl

Testing

Recently on one project I tried to execute the Zend Unit Test Suite, but stumbled on a memory exhaustion problem. If I tried to set the PHP memory limit above 2GB, my computer crashed, probably when it tried to swap. I realized that if the 2GB memory on my desktop machine is insufficient, the 512MB on the cruisecontrol machine will never be enough for running the whole test suite.

On the CMAX.gg -project our coders implemented a custom shell-script based cruisecontrol system, that ran each SimpleTest test case through the web browser. On the current project I decided to use open source tools such as phpUnderControl and PHPUnit instead, so that I could use a readily available tool instead of trying to parse a new tool from the scratch again. Also, it seemed that the PHPUnit is more active and has nowadays more support than the SimpleTest.

After configuring phpUnderControl for several days, I realized that our old system was actually better than what is currently available. For example you need to restart Cruisecontrol.org Tomcat webapp each time you make changes to your config files, I needed to create separate scripts to check and restart Cruisecontrol when it crashes about once per week, and it uses a huge amount of memory and disk-space. I managed to reduce the disk-usage by disabling the phpDoc -target on each build. Yet more disappointments came when I realized it is not possible to run a single test case through the Cruisecontrol/phpUnderControl -interface and the graphs and the reports were almost unusable. If you are aware of the Agile practices, you really want to have only an overview, where you see the number of failed tests and the total number of tests. The phpUnderControl doesn’t sort the tests failed first unlike our own system does. I also realized already 5 years ago, that it is quite useless trying to draw a graph where you have the ratio or percentage of failed tests, since quite quickly you end up drawing a line with 99.9% and 99.8% pass ratios. A better graph is the absolute number of failed tests.

After Googling a bit, I ended also up to the PHPUnit Changelog, that advertised a new –process-isolation switch. The drawback was that I needed to upgrade from PHPUnit 3.3 to 3.4. After some experimenting I managed to run the tests, but they all failed to an unspecified error at an unspecified location like this:

124) Test_Ext_Zend_Zend::testSetState
RuntimeException: Warning: require_once(trunk/ext/Zend/tests/Zend/AllTests.php): failed to open stream: No such file or directory in trunk/test/Test/Ext/Zend/Zend.php on line 5
Call Stack:
0.0090    1206200   1. {main}() /home/pharazon/workspace/trunk/-:0
0.0093    1223456   2. require_once(‘/home/pharazon/workspace/trunk/test/Test/Ext/Zend/Zend.php’) /home/pharazon/workspace/trunk/-:5

As you see, the stack trace says that the error happened at an file called “-”, a quite hard trace to debug. After about a day of debugging I managed to get error messages to show by printing them out from the PHPUnit/Framework/TestCase.php ( print_r($job) ). After this I managed to figure out that PHPUnit creates dynamically a new PHP -file from a template for each TestCase. The basic Template is located at PHPUnit/Framework/Process/TestCaseMethod.tpl.dist . After more debugging, I managed to get most of the tests working by modifying the template and adding a bootstrap -file to setup the environment and the constants. PHPUnit –process-isolation uses reflection, var_export() and the magic __set_state() -method to serialize the TestCases on the fly. One hindrance was that the Zend -classes didn’t implement the __set_state() -method, so I was required to add them. The hardest ones to decipher were the Zend Translate -related classes, where just calling the constructor wasn’t sufficient. Finally I managed to create a function like this:

/**
* Magic reflection function to unserialize var_export’ed objects
* to be used with phpunit –process-isolation
* See more from http://www.thoughtlabs.com/2008/02/02/phps-mystical-__set_state-method/
* @param the same than for __construct()
* @author Pharazon
*/
public static function __set_state($data) {
return array_shift($data);
}

Compared to our own CMAX Cruisecontrol, it seems that the PHPUnit –process-isolation is a quite cumbersomely implemented and restricted. For example the current system doesn’t still enable running the tests on several cores, but only one thread is being used. Our Apache -based test runner was already 5 year ago able to run a multiple test cases per machine, and actually also distribute the running of tests across multiple test nodes. It’s quite easy through a http-interface, when Apache takes care of the multi-threading. Additionally the old system submitted the results on a test database. I think PHPUnit can do it also, but haven’t implemented it yet.

Next I started to package the solutions, but realized that using a custom statically modified PHPUnit and Zend was not probably the best idea in the world. Luckily I stumbled on the TestCase->prepareTemplate -function and realized it would be easy to create a subclass MyTestCase extends PHPUnit_Framework_TestCase that would implement the modifications without need for altering the PHPUnit source code statically. There are two changes required. The first problem is the loading of the bootstrap. The default template checks the loading of the file defined on the –bootstrap -command line parameter after importing of the globals. The problem was that the GLOBALS included the Zend -classes that were not yet introduced, since the bootstrap was not yet loaded. To solve this chicken-egg -problem the only solution was to create a custom TestCaseMethod.tpl that had a static bootstrap loader before introducing the GLOBALS. Additionally I added a new parameter called ‘bootstrap’. Here is the new MyTestCase:

<?php
/**
* My Test Case that extends PHPUnit 3.4 TestCase, featuring
* – custom preparation to support bootstrapping with –process-isolation
* @author Pharazon
*/
class MyTestCase extends PHPUnit_Framework_TestCase {

/**
* Performs custom preparations on the process isolation template.
*
* @param PHPUnit_Util_Template $template
* @since Method available since Release 3.4.0
* @author Pharazon
*/
protected function prepareTemplate(PHPUnit_Util_Template $template)
{
//use custom process-isolation template
$template->setFile(ROOT.’/test/MyCaseMethod.tpl’);
$template->setVar(array(‘bootstrap’ => ROOT.’/test/testBootstrap.php’));
}
}

Then I changed all tests to extend MyTestCase instead of the PHPUnit_Framework_TestCase. Here is also the modified MyCaseMethod.tpl template:

<?php
set_include_path(‘{include_path}’);

//require_once ‘{bootstrap}’;
require_once ‘{filename}’;

function __phpunit_run_isolated_test()
{
$result = new PHPUnit_Framework_TestResult;
$result->collectRawCodeCoverageInformation({collectCodeCoverageInformation});

$test = new {className}(‘{methodName}’, unserialize(‘{data}’), ‘{dataName}’);
$test->setDependencyInput(unserialize(‘{dependencyInput}’));
$test->setInIsolation(TRUE);
$test->run($result);

print serialize(
array(
‘testResult’    => $test->getResult(),
‘numAssertions’ => $test->getNumAssertions(),
‘result’        => $result
)
);
}
{globals}

__phpunit_run_isolated_test()
?>

I was happy when I got all the tests to pass :). But only for a moment…. Next I tried to get the tests to run on the CruiseControl / phpUnderControl -instance, but noticed that the PHPUnit 3.4 doesn’t support anymore PMD or the CodeCoverage -tools, but they have been extracted to separate processes. Doh! But it’s yet a new story that you will get later :)

Published by: