Category Archives: Software Engineering

Jenkins git remote slave problem

Linux Software Engineering

Today I came up with an issue with Jenkins Continuous Integration (v.1.412) server when I was trying to fetch a git repository on a remote slave server (Red Hat). I found a few similar issues and open bugs, but none of them were directly related to Linux slaves. I got this error message dump:

Started by user jenkins
Building remotely on server01
Checkout:MyProject / /home/jenkins/workspace/MyProject – hudson.remoting.Channel@5352c503:server01
Using strategy: Default
Last Built Revision: Revision ebb0c40a1a321a00d8176e25aa81364efaac702f (origin/master)
Checkout:MyProject / /home/jenkins/workspace/MyProject – hudson.remoting.LocalChannel@59ab12f8
Fetching changes from 1 remote Git repository
Fetching upstream changes from ssh://git.server/var/repos/git/MyProject
ERROR: Problem fetching from origin / origin – could be unavailable. Continuing anyway
ERROR: (Underlying report) : Error performing command: git fetch -t ssh://git.server/var/repos/git/MyProject +refs/heads/*:refs/remotes/origin/*
Command “git fetch -t ssh://git.server/var/repos/git/MyProject +refs/heads/*:refs/remotes/origin/*” returned status code 128: error: cannot run ssh: No such file or directory
fatal: unable to fork

ERROR: Could not fetch from any repository
FATAL: Could not fetch from any repository
hudson.plugins.git.GitException: Could not fetch from any repository
at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1008)
at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:968)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:1956)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at hudson.remoting.Engine$1$1.run(Engine.java:60)
at java.lang.Thread.run(Thread.java:636)

The issue was resolved by adding correct location of the ssh -command (on my server /usr/bin) to the PATH of slave node’s environment settings (on Jenkins).

Published by:

How to integrate CakePHP to Joomla 1.6 as a component

Coding Software Engineering

Have you ever dreamed about integrating Joomla and CakePHP together such a way, that the best sides of both the systems would be utilized most efficiently? We’ll tell you now, how to do that!

Like we know, CakePHP is an excellent tool (for example) creating CRUD-interfaces from existing databases in a quick and easy fashion. However, it often takes too much of work when you have to code some features again and again (like image galleries, user registration system and many other things). On the other hand, Joomla is a well known content management system, that has a lot of features and plugins that are helpful when creating functional, dynamic and good-looking websites. So is there a way to make Cake and Joomla to work together?

Some years ago there was published an article about integrating Joomla and CakePHP together (this integration was called “JAKE”). As many people (including we) found this article very useful, and since both of the systems have evolved during last years, we thought it is now a right time to update article’s technical content to cover Joomla 1.6 and CakePHP 1.3. Like in the article mentioned above, also this article is meant to inform you how to include CakePHP to Joomla as a component.

1. Start by installing CakePHP 1.3 in ‘\components\com_cake’ of Joomla 1.6 directory. Of course, you need to create the subdirectory ‘com_cake‘ first. When CakePHP is installed, the directories app, cake, plugins and vendors should be found directly under ‘\com_cake’ (like com_cake\app etc.).

2. After installation you have to create the so-called triggers for the component. Without them, Joomla can’t use the component. So open your favourite text-editor and create files “cake.php” and “cake.html.php” to ‘components\com_cake‘ -directory. Put following content to the files:

cake.php:

<?php
defined( '_JEXEC' ) or die( 'Restricted access' );

require_once( JApplicationHelper::getPath('front_html') );
jimport('joomla.application.component.controller');
jimport('joomla.application.component.helper');

$document = JFactory::getDocument();
$document->setTitle("com_cake: Ultimate Joomla Component");
$joomla_path = dirname(dirname(dirname(__FILE__)));

// As this component (cakephp) will need database access, lets include Joomla's config file
require_once($joomla_path.'/configuration.php');

// Constants to be used later in com_cake
$config = new JConfig();

define(JOOMLA_PATH,JURI::base());
define(DB_SERVER,$config->host);
define(DB_USER,$config->user);
define(DB_PASSWORD,$config->password);
define(DB_NAME,$config->db);

$controller = JArrayHelper::getValue($_REQUEST ,'module'); //option passed is treated as a controller in cake
$action = JArrayHelper::getValue($_REQUEST ,'task'); //task passed is treated as a controller in cake
$param = JArrayHelper::getValue($_REQUEST ,'id');
HTML_cake::requestCakePHP('/'.$controller.'/'.$action.'/'.$param);

cake.html.php:

<?php
defined( '_JEXEC' ) or die( 'Restricted access' );
class HTML_cake {
    function requestCakePHP($url) {
        $_GET['url']=$url;
        require_once 'app/webroot/index.php';
    }
}

Edit also Cake’s database.php following way:

class DATABASE_CONFIG
{
    var $default = array(
	'driver' => 'mysql',
	'connect' => 'mysql_connect',
	'host' => DB_SERVER,
	'login' => DB_USER,
	'password' => DB_PASSWORD,
	'database' => DB_NAME
    );
}

3. For next, let’s configure rewrite-rules. Joomla will now take over the URL Rewriting for CakePHP. Disable all CakePHP .htaccess files by renaming them for example as “htaccess” (without the leading dot):

* components/com_cake/.htaccess
* components/com_cake/app/.htaccess
* components/com_cake/app/webroot/.htaccess

Configure then App.baseURL from ‘components/com_cake/app/config/core.php‘, by uncommenting and editing the following row:

Configure::write('App.baseUrl', 'components/com_cake/app');

4. After that, we need to modify Cake’s helpers that are used to output links, images and other things. This must be done in order to make the helpers to output URLs in Joomla’s format:

index.php?option=com_cake&module=names&task=edit&id=5

So open firstly the file ‘app/config/bootstrap.php‘ and add the following function to end of the file. It’s possible that you have to modify it afterwards (according to your application), but the function’s idea is to find individual parameters given by the user in URL (like “task=edit” etc.) and output the URL in Joomla’s format:

function reform_url($url) {
    // Explode url to parts according to the / -character
    $temp=explode('/',$url);
    $controller = false;
    $action = false;
    $param = false;
    if (count($temp) > 3) $controller = "&module=".$temp[3];
    if (count($temp) > 4) $action = '&task='.$temp[4];
    if (count($temp) > 5) $param = '&id='.$temp[5];
    $url=JOOMLA_PATH.'index.php?option=com_cake'.$controller.$action.$param;
    return $url;
}

5. Then integrate the function “reform_url” with all the helper functions that process URLs, to make them output URLs in Joomla’s format. Copy file ‘cake/libs/view/helpers/html.php‘ and ‘form.php‘ to directory ‘app/views/helpers‘ and file ‘cake/libs/controller/controller.php’ to ‘app/controllers‘. After that, edit the following functions:

html.php:

function link($title, $url = null, $options = array(), $confirmMessage = false) {
	$escapeTitle = true;
	if ($url !== null) {
           $url = reform_url($this->url($url));
	} else {
		$url = reform_url($this->url($title));
		$title = $url;
		$escapeTitle = false;
	}
	if (isset($options['escape'])) {
		$escapeTitle = $options['escape'];
	}
	if ($escapeTitle === true) {
		$title = h($title);
	} elseif (is_string($escapeTitle)) {
		$title = htmlentities($title, ENT_QUOTES, $escapeTitle);
	}
	if (!empty($options['confirm'])) {
		$confirmMessage = $options['confirm'];
		unset($options['confirm']);
	}
	if ($confirmMessage) {
		$confirmMessage = str_replace("'", "\'", $confirmMessage);
		$confirmMessage = str_replace('"', '\"', $confirmMessage);
		$options['onclick'] = "return confirm('{$confirmMessage}');";
	} elseif (isset($options['default']) && $options['default'] == false) {
		if (isset($options['onclick'])) {
			$options['onclick'] .= ' event.returnValue = false; return false;';
		} else {
			$options['onclick'] = 'event.returnValue = false; return false;';
		}
		unset($options['default']);
	}
	return sprintf($this->tags['link'], $url, $this->_parseAttributes($options), $title);
}
function image($path, $options = array()) {
	if (is_array($path)) {
		$path = reform_url($this->url($path));
	} elseif (strpos($path, '://') === false) {
		if ($path[0] !== '/') {
			$path = IMAGES_URL . $path;
		}
		$path = $this->assetTimestamp($this->webroot($path));
	}
	if (!isset($options['alt'])) {
		$options['alt'] = '';
	}
	$url = false;
	if (!empty($options['url'])) {
		$url = $options['url'];
		unset($options['url']);
	}
	$image = sprintf($this->tags['image'], $path, $this->_parseAttributes($options, null, '', ' '));
	if ($url) {
		return sprintf($this->tags['link'], reform_url($this->url($url)), null, $image);
	}
	return $image;
}

form.php:

function create($model = null, $options = array()) {
        $created = $id = false;
        $append = '';
        $view =& ClassRegistry::getObject('view');
        if (is_array($model) && empty($options)) {
            $options = $model;
            $model = null;
        }
        if (empty($model) && $model !== false && !empty($this->params['models'])) {
            $model = $this->params['models'][0];
            $this->defaultModel = $this->params['models'][0];
        } elseif (empty($model) && empty($this->params['models'])) {
            $model = false;
        }
        $models = ClassRegistry::keys();
        foreach ($models as $currentModel) {
            if (ClassRegistry::isKeySet($currentModel)) {
                $currentObject =& ClassRegistry::getObject($currentModel);
                if (is_a($currentObject, 'Model') && !empty($currentObject->validationErrors)) {
                    $this->validationErrors[Inflector::camelize($currentModel)] =& $currentObject->validationErrors;
                }
            }
        }
        $object = $this->_introspectModel($model);
        $this->setEntity($model . '.', true);
        $modelEntity = $this->model();
        if (isset($this->fieldset[$modelEntity]['key'])) {
            $data = $this->fieldset[$modelEntity];
            $recordExists = (
                isset($this->data[$model]) &&
                !empty($this->data[$model][$data['key']]) &&
                !is_array($this->data[$model][$data['key']])
            );
            if ($recordExists) {
                $created = true;
                $id = $this->data[$model][$data['key']];
            }
        }
        $options = array_merge(array(
            'type' => ($created && empty($options['action'])) ? 'put' : 'post',
            'action' => null,
            'url' => null,
            'default' => true,
            'encoding' => strtolower(Configure::read('App.encoding')),
            'inputDefaults' => array()),
        $options);
        $this->_inputDefaults = $options['inputDefaults'];
        unset($options['inputDefaults']);
        if (empty($options['url']) || is_array($options['url'])) {
            if (empty($options['url']['controller'])) {
                if (!empty($model) && $model != $this->defaultModel) {
                    $options['url']['controller'] = Inflector::underscore(Inflector::pluralize($model));
                } elseif (!empty($this->params['controller'])) {
                    $options['url']['controller'] = Inflector::underscore($this->params['controller']);
                }
            }
            if (empty($options['action'])) {
                $options['action'] = $this->params['action'];
            }

            $actionDefaults = array(
                'plugin' => $this->plugin,
                'controller' => $view->viewPath,
                'action' => $options['action']
            );
            if (!empty($options['action']) && !isset($options['id'])) {
                $options['id'] = $this->domId($options['action'] . 'Form');
            }
            $options['action'] = array_merge($actionDefaults, (array)$options['url']);
            if (empty($options['action'][0])) {
                $options['action'][0] = $id;
            }
        } elseif (is_string($options['url'])) {
            $options['action'] = $options['url'];
        }
        unset($options['url']);

        switch (strtolower($options['type'])) {
            case 'get':
                $htmlAttributes['method'] = 'get';
            break;
            case 'file':
                $htmlAttributes['enctype'] = 'multipart/form-data';
                $options['type'] = ($created) ? 'put' : 'post';
            case 'post':
            case 'put':
            case 'delete':
                $append .= $this->hidden('_method', array(
                    'name' => '_method', 'value' => strtoupper($options['type']), 'id' => null
                ));
            default:
                $htmlAttributes['method'] = 'post';
            break;
        }
        $this->requestType = strtolower($options['type']);

        $htmlAttributes['action'] = reform_url($this->url($options['action']));
        unset($options['type'], $options['action']);

        if ($options['default'] == false) {
            if (isset($htmlAttributes['onSubmit']) || isset($htmlAttributes['onsubmit'])) {
                $htmlAttributes['onsubmit'] .= ' event.returnValue = false; return false;';
            } else {
                $htmlAttributes['onsubmit'] = 'event.returnValue = false; return false;';
            }
        }

        if (!empty($options['encoding'])) {
            $htmlAttributes['accept-charset'] = $options['encoding'];
            unset($options['encoding']);
        }

        unset($options['default']);
        $htmlAttributes = array_merge($options, $htmlAttributes);

        $this->fields = array();
        if (isset($this->params['_Token']) && !empty($this->params['_Token'])) {
            $append .= $this->hidden('_Token.key', array(
                'value' => $this->params['_Token']['key'], 'id' => 'Token' . mt_rand())
            );
        }

        if (!empty($append)) {
            $append = sprintf($this->Html->tags['block'], ' style="display:none;"', $append);
        }

        $this->setEntity($model . '.', true);
        $attributes = $this->_parseAttributes($htmlAttributes, null, '');
        return sprintf($this->Html->tags['form'], $attributes) . $append;
}

controller.php:

function flash($message, $url, $pause = 1, $layout = 'flash') {

    $this->autoRender = false;
    $this->set('url', reform_url(Router::url($url)));
    $this->set('message', $message);
    $this->set('pause', $pause);
    $this->set('page_title', $message);
    $this->render(false, $layout);
}

Finally, edit the file ‘cake/libs/router.php‘:

function url($url = null, $full = false) {

        $self =& Router::getInstance();
        $defaults = $params = array('plugin' => null, 'controller' => null, 'action' => 'index');

        if (is_bool($full)) {
            $escape = false;
        } else {
            extract($full + array('escape' => false, 'full' => false));
        }

        if (!empty($self->__params)) {
            if (isset($this) && !isset($this->params['requested'])) {
                $params = $self->__params[0];
            } else {
                $params = end($self->__params);
            }
        }
        $path = array('base' => null);

        if (!empty($self->__paths)) {
            if (isset($this) && !isset($this->params['requested'])) {
                $path = $self->__paths[0];
            } else {
                $path = end($self->__paths);
            }
        }
        $base = $path['base'];
        $extension = $output = $mapped = $q = $frag = null;

        if (is_array($url)) {
            if (isset($url['base']) && $url['base'] === false) {
                $base = null;
                unset($url['base']);
            }
            if (isset($url['full_base']) && $url['full_base'] === true) {
                $full = true;
                unset($url['full_base']);
            }
            if (isset($url['?'])) {
                $q = $url['?'];
                unset($url['?']);
            }
            if (isset($url['#'])) {
                $frag = '#' . urlencode($url['#']);
                unset($url['#']);
            }
            if (empty($url['action'])) {
                if (empty($url['controller']) || $params['controller'] === $url['controller']) {
                    $url['action'] = $params['action'];
                } else {
                    $url['action'] = 'index';
                }
            }

            $prefixExists = (array_intersect_key($url, array_flip($self->__prefixes)));
            foreach ($self->__prefixes as $prefix) {
                if (!empty($params[$prefix]) && !$prefixExists) {
                    $url[$prefix] = true;
                } elseif (isset($url[$prefix]) && !$url[$prefix]) {
                    unset($url[$prefix]);
                }
                if (isset($url[$prefix]) && strpos($url['action'], $prefix . '_') === 0) {
                    $url['action'] = substr($url['action'], strlen($prefix) + 1);
                }
            }

            $url += array('controller' => $params['controller'], 'plugin' => $params['plugin']);

            if (isset($url['ext'])) {
                $extension = '.' . $url['ext'];
                unset($url['ext']);
            }
            $match = false;

            for ($i = 0, $len = count($self->routes); $i < $len; $i++) {                 $originalUrl = $url;                 if (isset($self->routes[$i]->options['persist'], $params)) {
                    $url = $self->routes[$i]->persistParams($url, $params);
                }

                if ($match = $self->routes[$i]->match($url)) {
                    $output = trim($match, '/');
                    break;
                }
                $url = $originalUrl;
            }

            $output = str_replace("/sort:","/sort=",$output);
            $output = str_replace("/direction:","/direction=",$output);

            if ($match === false) {
                $output = $self->_handleNoRoute($url);
            }

            $output = str_replace('//', '/', $base . '/' . $output);
            } else {
            $url = reform_url($url);
            if (((strpos($url, '://')) || (strpos($url, 'javascript:') === 0) || (strpos($url, 'mailto:') === 0)) || (!strncmp($url, '#', 1))) {
                return $url;
            }
            if (empty($url)) {
                if (!isset($path['here'])) {
                    $path['here'] = '/';
                }
                $output = $path['here'];
            } elseif (substr($url, 0, 1) === '/') {
                $output = $base . $url;
            } else {
                $output = $base . '/';
                foreach ($self->__prefixes as $prefix) {
                    if (isset($params[$prefix])) {
                        $output .= $prefix . '/';
                        break;
                    }
                }
                if (!empty($params['plugin']) && $params['plugin'] !== $params['controller']) {
                    $output .= Inflector::underscore($params['plugin']) . '/';
                }
                $output .= Inflector::underscore($params['controller']) . '/' . $url;
            }
            $output = str_replace('//', '/', $output);
        }
        if ($full && defined('FULL_BASE_URL')) {
            $output = FULL_BASE_URL . $output;
        }
        if (!empty($extension) && substr($output, -1) === '/') {
            $output = substr($output, 0, -1);
        }

        return $output . $extension . $self->queryString($q, array(), $escape) . $frag;
}

6. Your new CakePHP -component is now ready to be tested, so open it by writing an URL like:

http://localhost/joomla/index.php?option=com_cake

If everything went right, you’ll see Cake’s homepage in Joomla layout. Make sure you have edited your default.html file so that CSS and HTML tags do not mess up.

7. Maybe you want to distribute your new CakePHP -component for your friends or other people? Even if you don’t, it is recommended that all the components used by Joomla are installed correctly to CMS.

It’s quite easy task to to create an install package for your CakePHP -component. Just create an empty folder to somewhere on your disk (like ‘temp/com_cake‘) and make there two subdirectories: ‘site’ and ‘admin’. Then copy all the content from the folder in where you just created your component (like ‘joomla/components/com_cake‘) to the new sub-directory ‘temp/com_cake/site‘.

As we aren’t going to make an admin-interface for the CakePHP-component (at least, yet), just create the following two files to the folder ‘temp/com_cake/admin‘:

cake.php:

<html>
<body>
    CakePHP component v1.0
</body>
</html>

index.html:

<html>
<body>
</body>
</html>

Finally, create an install-file for the component. Open your favorite text-editor and create file ‘install.xml‘ to the root folder ‘temp/com_cake‘. Add the following content to the file:

<?xml version="1.0" encoding="utf-8"?>
<install type="component" version="1.6.2">

 <name>Cake</name>
 <!-- The following elements are optional and free of formatting constraints -->
 <creationDate>2011-08-08</creationDate>
 <author>Your name</author>
 <authorEmail>Your e-mail address</authorEmail>
 <authorUrl>Your web-site</authorUrl>
 <copyright>Copyright Info</copyright>
 <license>License Info</license>
 <!--  The version string is recorded in the components table -->
 <version>1.00</version>
 <!-- The description is optional and defaults to the name -->
 <description>CakePHP inside Joomla</description>

 <!-- Site Main File Copy Section -->
 <!-- Note the folder attribute: This attribute describes the folder
      to copy FROM in the package to install therefore files copied
      in this section are copied from /site/ in the package -->
 <files folder="site">
     <filename>cake.php</filename>
     <filename>cake.html.php</filename>
     <filename>index.php</filename>
     <filename>README</filename>
     <folder>app</folder>
     <folder>cake</folder>
     <folder>vendors</folder>
     <folder>plugins</folder>
 </files>

 <administration>
  <!-- Administration Menu Section -->
  <menu>Cake</menu>

  <!-- Administration Main File Copy Section -->
  <files folder="admin">
      <filename>cake.php</filename>
      <filename>index.html</filename>
  </files>

 </administration>
</install>

8. When you have saved install.xml -file, just pack it with the subdirectories “site” and “admin” to the zip-package ‘com_cake.zip‘. This file can then be distributed and installed to Joomla simply with using Joomla’s Extension manager!

Article written by
Jussi Laine

Published by:

It’s Not Your Fault – Blame the Designer

Software Engineering Start-up Testing

Most people don’t realize how often they actually do mistakes. Every wrong move or thoughtless action, even if you can recover it immediately, is still a mistake that could have been possible to perform flawlessly. In general, people don’t admit doing errors, especially publicly in fear of becoming shamed. Six Sigma suggests as the source of the most of the cost in the products the so called Hidden Factories, or error corrections that are performed by the employees without informing the management, to avoid taking the blame, but incurring nevertheless the cost of the error. However, the belief that the mistake that you just made would be your fault is actually not true! If you just made an error, it’s actually not your fault! Instead, you should point your finger to the designer.

At Toyota the Kaizen continuous improvement and learning organization uses the following framework for blaming and learning from mistakes. You can adapt this philosophy also on your personal life, but I discuss it from the entrepreneur’s point of view.

1. Blame the Instructions

If you don’t have instructions how to perform a specific task, the work design and management is to be blamed. The definition of the management and entrepreneurs is actually not do any work, but to manage the organization of work, meaning how exactly you should perform your tasks and which tasks should be performed in the first place. The work of an entrepreneur is to organize the factors of production so that you can convert ideas into practice filling out a market niche. The work of the managers is to find optimal organization of the work for a given problem. The worker should not be interested of the management of the work, since it’s not his primary interest, as long as the company stays afloat and employment is guaranteed. Similarly, a manager or entrepreneur who is not interested in the continuous improvement of the work processes is not performing his job properly.

The current and western management nomenclature tends to omit the first and primary factor of production as irrelevant – the standardization of work. However, the as any basic economics book can tell you straight-on, without standardization of work there is no economies of scale and the work remains ad hoc in nature and economically unfeasible. In the western countries the emphasis is put on the innovation, or the Deming’s PDCA -cycle, how to improve the current work process by radical innovation. Unfortunately most often the basics are forgotten in the fuzz of innovation and the management fails to standardize the work after or before the innovation, resulting in utter chaos and high costs. No wonder that the jobs are continuously being outsourced to China and India.

Secondly, if by a miracle a standardized work process would exist in your organization, the instructions might well be incomplete, inaccurate and ignorant of the most common error modes. The first corrective action at Toyota is aimed thus on improving the instructions so that the by following them the work can be performed succesfully.

Yet a more advanced philosophy for standardization is the Six Sigma -approach that tries to identify and mitigate the sole possibility of producing an error. The Japanese call this principle as Poka-Yoke, or error-proofing. The idea is that by eliminating the source cause for errors the work process can be performed flawlessly producing significant quality and cost improvements.

2. Blame the Machines

If despite improving the instructions you still make them, you must turn the look on the machinery and tools that you are using to perform the task. For example if your computer is crashing and preventing you from performing your work, it’s not the fault of the instructions or yourself, but of the computer. Thus you should find a new tool or machine design that makes it impossible to perform such an error.

The Designed for Six Sigma (DFSS) -method tries to achieve exactly this. 80% of the quality problems can be tracked back to the design board, and thus it is not possible to reach the higher levels of quality (5 and 6 sigma) without taking the high quality in account already on the drawing board. The DFSS tries to eliminate the potential for producing Critical-to-Quality (CTQ) error in advance by for example preferring usage of proven components, analysing the potential error modes (FMEA) and the Voice of Customer.

The Japanese have developed a 14 level model of intelligent machine automation (Jidoka). The idea is to convert the machines to automatically detect errors in themselves. The idea comes from Toyota’s founder’s Sakichi Toyoda’s automatic weaving machine that automatically detected a broken loom. Six Sigma suggests of using simple automatic pass/fail -gates for investigating the quality level on a production line. The same idea is called as Test-Driven Development in software engineering, where you before starting of the coding define the end product by writing automated unit and acceptance tests.

3. Blame Yourself

If you are not able to follow the instructions or the standardized process, it’s your fault. In this case Toyota considers a task for you that you are able to perform according to the instructions. However, before blaming on the human factors, one should consider the two previous steps if they have rendered the work inhumane to perform, and if the blame can be still be placed on the Work and Machine Design first.

Of course, if you are unable to follow the Kaizen continuous improvement -cycle, the blame is truly only on yourself :)

Published by:

The Inmates are running the Agile Asylum

Management Software Engineering Usability

Here is a great interview of Alan Cooper, the designer of Visual Basic & Visual Studio and the author of the Goal-Driven Design (GDD), a close relative to the GUIDe by Sari A. Laakso:

http://www.infoq.com/interviews/Interaction-Design-Alan-Cooper

Agile is not about productivity, it’s about the core of motivation of the developers. The traditional management of the software projects usually lack understanding what is going on, set unrealistic objectives, drive for low quality and make the work of the developers miserable. While the industrial-era management doesn’t really understand what’s going on, the developers have filled out the vacuum by managing themselves. Instead, the knowledge workers are not motivated by money or following the schedules, but doing good (or great) work. Thus the rise of the Open Source, over there people can do as good products as they wish :) If the industrial management techniques makes this impossible by managing the knowledge workers as industrial workers, then the developers are not pleased either. Because nobody is really managing the development work, the Agile has risen from the ranks of the developers to fix the management problem.

Alan Cooper claims that the process of the interaction design is quite similar to the agile ways. The key process is to reflect on the business problem before day zero of the start of the development. The key thing is to have deep and profound understanding of the business process.

Tasks are not Goals

The important thing is not what tasks the users do, but what is the end state. The design process is to redesign the tasks so that the goals can be achieved easier. By designing based on the tasks the result will be a Dancing Bear :). It dances, but not very prettily. The objective of the interaction design -school / GDD / GUIDe is to make the bear to dance well!

In software there is no economics of scale

In the old times the main driver of the business was to get the unit costs down. However, in the software industry the maintenance costs are zero or very low, but the development cost is rather high. The economics of the software are profoundly different to the economics of the manufacturing. Driving the cost down just drives down the desirability of the design. The most important thing is to worry how you can elevate your number one goal. The business managers should think only about how to increase the business value and quality instead of reducing the cost.

Published by:

Iterationless Scrum III

Software Engineering

I found a second article describing an iterationless agile SW process. This time it’s Dan Rawsthorne from Danube introducing the new Scrum III. He says that the Sprints should not be anymore strictly obeyed, but the user stories can enter the WIP (Work-in-Progress) -queue accriss the Sprint boundaries!

It seems that he wants to move Scrum more to the direction of the Lean by replacing the Sprints by an iterationless Kanban/Just-in-Time pull-system. I found already one article earlier about the same idea. For me this seems rational, since the original Lean really actually has no iterations, they are just remnants of the legacy SW Engineering idea of Iterative and Incremental processes. A true Lean process is always one that releases one feature at a time, with a constant pace, regularly, and carries no Technical Dept or half-finished features. The optimum way is to use one-piece-flow meaning working only on one job at a time. Multi-tasking has been scientifically been proven to be sub-optimal since you introduce waste by increasing the unnecessary waiting time, making the cycle times slower and thus increasing the inventory / WIP. If you don’t release immediately when your feature is completed, you essentially store your code in a non-productive warehouse. You have paid the salaries, but haven’t earned a cent out of the investment. The situation gets even worse if you have happened to code some bugs in – the waiting time before release increases the cycle time for finding and fixing the bugs. The faster the feedback cycle time, the better, thus you should release in small batches and immediately when you have completed some code worth of business value. In the Extreme Programming the 2-week fixed iterations are suboptimal in this sense, although you have the Release Often -rule. However, working on two or more user stories simultaneously increases the real cycle time and thus cost.

I’m not currently managing any SW Engineering team, but an iterationless XP/Scrum is definately something I will use on my next project. The only thing to do is to work only on one user story at a time. Shouldn’t be too hard to get it to work :).

Published by:

Network Centric Software Engineering

Software Engineering

I was reading the Cooper Journal of Interaction Design when I stumbled on a few articles on Sensemaking and Network Centric Operations (NCO), a new military organization doctrine that tries to use information technology to facilitate operations of geographically dispersed units and to enhance operational effectiveness compared to a traditional force. For me this sounds relevant to software engineering, too, so I got interested. Wikipedia noted that a similar approaches have already been adopted by the UK and Swedish military.

The idea of the NCO is to enhance the situational awareness by improving the information exchange between the troops. This includes perceiving the environment, comprehending its meaning, and projecting the status in the future.  As a software engineer this sounds very familiar. The SW projects rarely have situational awareness even when all the developers are present on the same floor or even when they are sitting side by side. When the project is introduced with the off-site customer or the outsourced Indian developers, the information exchange problems tends to escalate even further. Why it is so difficult for the software engineering teams to exchange information even between the developers? One article on the Cooper Journal discussed the similarities between the SW Engineering and Movie production, but one commentor noted that the most significant difference is that in the movie casting the director has the final say of everything, while in the software projects nobody has the technical and visionary ability to comprehend all parts of the system or communicate on all topics.

The NCO approaches the problem differently. The key process of sensemaking involves all levels and perspectives of the organization and is grounded on a robustly networked force that is capable of information sharing. The second tenet assumes that the information sharing and collaboration enhances the quality of information and the shared situational awareness. This leads into self-synchronization of the units, that leads finally into a dramatical increase of mission effectiveness.

The sensemaking process by the U.S. Department of Defence starts by

  • Forming an awareness of key elements relevant to the situation. This entails knowing “the who, what, when and where.”
  • Forming an understanding of what it all means in some bounded context, based upon past experiences, training, education and cognitive capabilities. This entails:
    • Forming hypotheses and making inferences, i.e. generalizations (predictions or anticipations) about future events.
    • Forming a sense of the implications for different courses of action.
  • Making decisions by:
    • Generating alternative response actions to control the situation.
    • Identifying the objectives, constraints, and factors that influence the feasibility and desirability of each alternative.
    • Conducting an assessment of these alternatives.

The enhanced collaboration and higher quality of information enables people to make better decisions. I can see directly, how this can be applied on a distributed software engineering project. The practices such as a Daily Scrum already implements partly the same purpose, but it should be taken further. I envision that a NCO software engineering team would utilize tools like Twitter to continuously update the status of what they are doing, what are their problems, what things should be known about and estimates about the remaining effort. Yesterday I watched a Apollo 11 document where the lead flight director Gene Kranz explained how he simultaneously listened to 7 simultaneous communication loops, but with training was able to focus on the one communication that was important for performing the current task. Similarly the twits could be a way to exchange information and meaning in a distributed SW project. However, a key enabler is to identify the communication language, or the common concepts everybody should understand. Another point is to define the key elements of the situation. The NCO -model seems to promote also the learning organization -dogma by giving permission and tools for the team members to understand and challenge the  knowledge of others (or the situational awareness). Often the true ground-reason for project failures is not the tools or the people, but the inability to acknowledge the current situation and react to the changes.

The IT industry is surrounded by all kinds of communication software, but still unable to utilize or leverage it for meaningful communication. Perhaps the problem is with the cognitive models of software team organization and new models such as the NCO is desperately needed.

Published by:

Lean & Agile Software Engineering

Software Engineering

I got recently my second thesis work completed, during which I managed to read quite a lot management literature from the 80′s and even the magnificent Henry Ford’s “My life and work” from 1922 describing most of the Total Quality Management principles almost 90 years ago. Yet we are still not using the practices that produced one of the greatest industrial enterprises to the date.

Another fundamental and nicely written book is Eliyahu Goldratt’s The Goal describing the management process of Theory of Constraints. According to Goldratt, the mission of a company is to maximize the throughput (sales to the customers), while simultaneously minimizing the operational expenses and the inventory. Every system has a single bottleneck that constraints the throughput. By locating the bottleneck one can improve the throughput, but improving any other area will not improve the performance at all. This principle is the same as the Extreme Programming’s Leave Optimization till last. Otherwise you end up in tuning parts of your code that will not improve the performance at all, but make your code harder to understand.

After reading these and a few more books the agile software engineering principles suddenly appeared much clearer and theoretically more worthwhile. I began to understand the fundamental principles behind the methods like Scrum, XP and Lean, and learned a few new tricks that haven’t been used much before in SW engineering. Unfortunately Kent Beck hasn’t bothered to reference to the prior work and many of the Agile and Lean principles have been challenged and questioned, while their origins have remained unrecognized. For example the principle of quick iterations can be derived directly to the Goldratt’s production formula: the faster you can release the less Work-In-Progress or inventory you have. The less inventory not currently producing revenues, the lower are your capital costs in holding the inventory and the higher profit you have. Additionally you get also the feedback faster, which is a crucial meta-capability in the era of tightening competition between the Lean organizations (according to Robin Cooper: When Lean Enterprises Collide). Actually, when you think about it, the optimum iteration length or batch size is one or release cycle time close to zero, or in other words you could even use iterationless-process or a Drum-Buffer-Rope/JIT -pull-system as described by Goldratt.

A related building block of a JIT -production system are the Kanban -signal cards, that are closely analogous to the XP User Stories and the Scrum index cards. The idea is to pull the production from the customer/demand side by signaling the previous phases of the permit to produce new items, instead of producing documentation/code that might not be needed by anyone. The idealized model is the Just-in-Time -system, where a customer gets what he orders when he needs it, but nothing is produced in advance on stock. The magic in the system is the process capability, or the variance and the tolerance you need to have spare stock so that the sudden demand spikes won’t cause bottlenecks (or the increase in the cycle time lowering the critical throughput). According to the Deming’s Statistical Process Control -school, a process capability means how low variance the process has, or how low emergency stock you can keep without causing a bottleneck. A SW Engineering example of this is for example how many XP User Stories you have to produce before the beginning of an iteration, or can you also produce them Just-in-Time. At least on my XP team this reveals directly one bottleneck: we do have a queue of roughly half-foot of User Stories and the engineering team seems to be much slower to implement the stories than the customer is able to generate them. Also Goldratt notices that the easiest way to identify a bottleneck is by looking for the queue sizes, the longest queue is usually the bottleneck, especially if the queue seems to be always increasing.

From the process measurement point of view XP seems to draw the principles almost directly from the text book when introducing the Project Velocity -measurement – the only required measurement for a software team is its throughput, or how many user stories or story points they can release per iteration. Unfortunately Beck doesn’t explain very well why this measurement is the most important one, but you get it now: the most severe bottleneck in SW engineering is the coding, not the design – so you should first focus on getting your tput up. There is so much to do in improving the velocity that I don’t know yet where the next bottleck will be. I think it will take quite and much work before anyone can say that they have more designers than coders and are able to locate the next major software engineering bottleneck :).

When you know the TQM -fundamentals you will also know the remedies for the performance and quality problems. The mission of the TQM -school (as first stated by Henry Ford) is to eliminate all waste. Somebody asked from me the what is the difference between the German quality and the Japanese quality? The difference is in the principle of eliminating the waste. When a car comes out from a German car factory, it goes first to a repair shop, and then to the customer. A Japanese car can be shipped directly to the customer. The difference is in how much scrap and rework you produce. As anyone in the software industry can easily recognize, the complete rewrites and bug fixes are the activities that most of the developers spend most of their days on producing also most of the cost. The Six Sigma -school goes even further and describes the Hidden Factories meaning all scraping and rework activities that are performed without appearing on the management reports. A good example is a bug that is detected and fixed immediately without entering it on the bug management system. This happens every day. One could think that a hidden rework factory is also a coding typo that requires use of backspace. According to the Japanese concept of Poka-Yoke, or error-proofing, the remedy is to use tools and practices that make it impossible to produce an error. For example you can use Eclipse that automatically displays the syntax errors, highlights the code, suggests for the possible (and correct) methods available, and autofills the brackets so that you can’t produce the elementary coding mistakes. Software engineering is full of social practices such as code reviews and pair programming that tries to further improve the quality by detecting the errors faster. However, practicing the Poka-Yoke (or the original Baka-Yoke = idiot proofing) will get you farther more quickly when you dismiss the possibility of producing errors in the first place! Yet more examples of the same principles include the Japanse concept of Jidoka- or a self-correcting machine that halts it’s operations and alerts a maintainer, when an unrecoverable error happens, and the Designed for Six Sigma, where product is designed by choosing components that are known to have low number of errors instead of coding all code from the scratch by self.

After introducing yourself on the key Lean/TQM concepts you should hopefully pretty quickly start to recognize from which more fundamental principle all the practices originate from, and in which situations they should and should not be used. I hope that with these few advices you can start to recognize where your bottlenecks currently are located in, and also get ideas how to improve your throughput and quality :). I’ll be also writing more on the topic :)

Published by: