Reflections on enterprise Agile

or...
why I hate "Agile" as it's perceived today
and what does Agile mean to me

I'm a well-known anti-Agile pundit when it comes to enterprises: sometimes I literally can't help to not ridicule people taking it seriously. I'd like to state the reasons why.

First off,

• I'm not against seriously taken processes at all.
  
  
• I'm not against something which I call Agile.
  
  
• The best time of my professional life was at a team doing SCRUM (more on that later)
  

In the late 90s, software development was in a - usual - crisis: projects didn't get done. There were a lot of people praising about how RUP or any other process would be better, and they tried to fulfill process criterias instead of solving end-user needs.

People simply had enough of this: they wanted to improve the World.

Agile manifesto was born.

Nowadays, I hear the following excuses:

• No, we can't ship this to you until End Of Sprint
  
• No, we cannot give your manager a preview of functionality, he can see it on the Sprint Demo day (editor's note: 1 day before deadline)
  
• No, we can't commit this bugfix yet as we didn't find out how to test it, even if it's obvious it works - we cannot ruin our test coverage
  

To the contrary:

• We don't have meeting notes, we're agile (not to speak about action points)
  
• We don't have formal inter-team interface definitions nor formalized technical (not UX, not API doc) specifications
  
• Each day we have a standup (10-20 minutes), we have a meeting day each sprint, a demo, a research day demo, a planning and a retrospective meeting
  

Also, "we" deliver each sprint - except the term "delivery" for me has much more means than pure deployment of a new version which passes all the active tests.

I mean, seriously: what is it, if not processes over people?

It's just an example, albeit a pretty serious one; my other favourite is non-colocated standups over phone, especially when it's made sure that people have distinct set of capabilities and they don't want to touch each other's stuff anyway; or when it doesn't matter if we advance or not.

Agile, for me, was some kind of rebellion against a world with no sense, like this one.

One should recognize, that a large organization needs processes; we're only free to choose which one we implement in an organization. Some roles are built-in: you could call your managers product owners, if they still do sign your timesheets. Make sure that your way of thinking has respect of your realities.

It's strange that all project methodologies seem to become famous on what they never wanted to be:

• Waterfall never ever said you shall sequentialize modern sized-projects as a single chunk (a few thousand lines in its age were full programs, while I commit thousands of - hand-typed - lines on some more productive days),
  
  
• RUP never ever said you shall have 94 roles and to make sure you fully document everything (and perhaps the brilliance behind RUP will once again regain the enterprise space where it belongs)
  
  
• Agile never ever said you should follow process packages, especially not for the sake of following process packages.
  

Being or not being Agile is not the problem, especially when it comes to following or not following certain practices. These are tools. Not always applicable. If your problems don't fit, if these are not your problems, or if they won't work in your environment, don't use them.

Being Agile for the sake of being Agile is the least Agile thing you could ever-ever came up with.

Tutoring: Working with Students

I was tutoring for years, and if there's something I miss at my current workplace is the students around me: they were an inevitable part of my teams since january 2005, until I left EPAM last summer.

(Please note: these guys were mostly at their first or second workplace, and still doing university: in Hungary, it's common that young software engineers are taken as full- or part-time employees while still studying.)


Working with students is hard. It's hard because even if they know the theoretical answers, and didn't forget them right after passing the exams, it's not yet burned into their brain with "learning-by-doing". They're not confident enough to work in an elegant way.

The Beauty of MVC

The widely-known Model-View-Controller pattern was published first in 1988 (by Krasner & Pope). It was originally indented to represent user mental models and so on, and while some people may want to superseed it with new paradigms, today I want to talk about its beauty.

Because MVC not [only] models the user's mental model, it models the world.

It's nothing else, than a practical implementation of Hegelian dialectic.

Let's start with a simple scheme. The user sits besides a computer. [S]he looks at the monitor, while trying to achieve something by pressing buttons on the keyboard, or trying to wander around with a mouse or on a touchpad.

Generally, we don't really distinguish between the different input peripherials with MVC. The monitor is the output, other human-computer interface peripherials are for input, it's that easy:



(Based on image of flickr user ntr23 under Creative commons license)

A consistent naming scheme proposal for Javascript callbacks

Basically, there are two schemes of javascript callbacks:


function(other_params, onSuccess, onError){
}


and

function(other_params, callback){
}


where callback is called with:

callback(null, params..)

on success, and


callback(exceptionobject)

on failure.

What if we would choose the first one, but instead call them differently?

function(other_params, _return, _throw){
}


Imagine the body:

function(other_params, _return, _throw){
 var params;/*...*/
 some_other_async_operation(params, function(result){
    return_(result);
 }, function _catch(e){
    throw_(e);
 } 
}

Is it more clean what's happening here?

I came to a level, with some helper functions:

(function(x){
    var return_ = arguments.callee.getCallback(arguments);var throw_ = arguments.callee.getErrorCallback(arguments);
    var result = {};
    try_(result, "=", db.view.call_(db, "xxx/xxx", {startkey:x+"_", endkey:x+"_z", group:true }))(function(){
        console.log("success");
        console.log(result.length);  
    }).catch_(function(exception){
        console.log("error");
        console.log(exception);
        throw_(exception);
    }).finally_(function(){
        console.log("returning");
        return_();
    })();
})(10);

Opinions, ideas?

Software and Design

Recently we have seen a rise of "designers" in engineering practices, particularly software engineering. While visual designers and artists were always present in every engineering field - the frescos weren't painted by the architects of the basilicas most of the time - it's strange to see opinions of software engineers taken as "they don't understand design anyway"

Because software IS design.

A software in itself is nothing more than a plan, an explicit, consistent set of thoughts which we give to the computer to build from. The masons of the virtual dreamcastles aren't us, the programmers: we merely give the plans to the computer to build it from bits and bytes as bricks once turned on.

Our thoughts are explicit, because the computer can think only one way.

Our thoughts are consistent, at least, from the computer's perspective. Non-consistent thoughts get a so-called syntax error, and would never start on the user's machine. Any inconsistency means the literal crash of the program.

Design in software engineering is the blood-swearing process of getting an idea consistent and in harmony to its environment.

To the contrary, creating some nice buttons in our terminology is NOT considered design: one of my former bosses called it "masturbation". Creating buttons which are consistent and are in harmony with the user's train of thought and of the application, simplified to the end, but not a single step further IS design, and it's really hard work.

What surprises - and saddens - me if I get a "design" document from a designer which clearly wasn't distilled this way what we call design.

Claude E. Shannon - father of information engineering

If we speak about people who shaped informatics / software engineering, we cannot forget "the father of information age", Shannon, who died nearly exactly 10 years ago, on february 24th, 2001. One of his writing was claimed the "Magna Charta of the Information Age" - and probably this essay shaped our present much more than the now 10 years old other "Magna Charta".

Without him, we may not be able to listen to CDs or MP3s, or make phone calls on our mobile, let alone connect to a WiFi network.






But who was this guy?

The case of the wise flamer: In Memoriam Edsger W. Dijsktra

If you go to my desk at my current workplace, you'll find a bunch of books on top of each other. As of today, the topmost of them is a hungarian edition of Structured Programming from Ole-Johan Dahl, Edsger W. Dijsktra and Sir C. A. R. Hoare, which was published originally in 1972, although the articles in it are much more older.




Most people would have think that a book from 1978 has nothing to say about our current affairs. Nothing could be further from the truth. In fact, those things which stand for nearly 40 years might be the ones that are universal to our profession. Do believe me, a lot of things do!

On one side, Dijsktra could easily be one of the greatest flamers you've ever met. It's enough to look at this article about programming languages from him - don't forget he got a Turing award for his achievements in that field! - to know what I'm speaking about.

Most of people will remember him by the Dijsktra Algorythm, maybe the disappereance of the Goto statement from modern programming techniques. Not that subsconciously we wouldn't use a lot of his findings when we're writing programs.

One of his greatest misunderstandings I believe was about his rejection of versions, and version control systems. He said in an interview in 2001 that "[programs have version numbers] even with decimals like version 2.6 or 2.7, that's nonsense; while version 1 should have been the final product". He didn't see the need of tracking changes of a software, as they were to be implemented only once, perfectly.

He even wrote his articles this style: on paper, by hand.(In fact, he used a kind of blogging sometimes I'm tempted to use this way :) He gave photocopies of these handwritings out to friends and collegaues, who may have distributed it themselves.

Never forget that he was the strongest advocate of TDD in the early 70s, when Kent Beck, the inventor of unit tests was most probably still occupied by more childish stuff (born in 1961.)

Still, I do believe that his mistake was based upon the perception that customers, or whoever is to order the software from computer scientists - he used this title consequently - had a perfect, clear idea of how the product should look like, even if expressed vaguely. Maybe later we will be able to return and reveal the truths behind this platonian thinking, still it's probably not the way most our industry experiences software specification on a day-to-day basis.

(I do believe this thought has some base to stand, although I feel myself too weak and early to prove its pillars unfortunately)

Another of his mistake was that he didn't count on the externalities: it may be that the environment of the system changes aggressively in the lifetime of the product, wether it be development or production phase. For new needs, one must respond with a set of changes, the quickly the better (or at least faster than the competition does).

Also, it may be that our users have different needs than we may have thought: although we have scientific measurements to prevent this, as with social sciences, there's always a large area of mistake or unexpected things to happen.

His style sometimes makes it harder to understand the deep messages behind his writings: truly a genius of his age, Dijsktra was able to form our just-born-then industry, yet his findings may have much more long-term effect than we would be able to perceive, even today.

So, go and find some of his influental papers and interviews, maybe even those linked in this article. Then think about it: what if the really important things are to stand regardless of the centuries, even if a flamer says them first?

FPA: an old, forgotten method for estimation

(Originally comment on InfoQ)
I don't really use new stuff which has an "agile" vignette on them even in an agile environment.


An FPA estimation spreadsheet


In the most agile team I had, a real agile one (own company, only developers, own product, full commitment, cycles and so on) - and even afterwards, I used a modified version of the Function Point Analysis.

What does it say?
- Every user story will sure have some datasources to get data from. Say these are Input Forms (IFs). Let's say creating such a form takes 5 points.

- Every user story will surely have some outputs. Let's call them Output Forms. Let's say it takes 4 point to create them

- Every user story will have to deal with some internal datastructures - tables, files, so on. Let's see how many distinct classes (tables, filetypes) seem to be involved; let's say developing handling of such a type is about 9 points, and call them Internal Logical Files (ILFs)

- Some user stories will have to deal with systems not within reach; these could be unknown libraries, web service APIs, maybe even the mailsystem. Let's say handling such stuff is 12 points.

Then, for each of the stories, we have a formula, like:
Creating a login screen = 1 IF, 1 OF, 2 ILF, 0 EX

Add them together multiplied:
1 x 4 + 1 x 5 + 2 x 9 + 0 x 12 = 18 FP (function point)

We could have also additional measures, but this will make them ok.

Now we have to determine how much time does it take to create a single function point. It takes a lot of efforts; my agile team had about 0.5 hours, the enterprise team was sometimes as slow as 2 hours. So, creating a login form with all bells and whistles takes about a day, or a week, depending wether you are in a small, enthusiastic company, or in an enterprise cubicle. Makes a lot of sense, even if you don't like the results.

Also the factors could be changed.

It's not agile, but this method was able to provide +-10% efficiency; so, after the second iteration, it's expected that a two week sprint's products cannot be late more than a day.

Archive: Software Engineering as an engineering discipline

Cross-posted from old blog

I have told this multiple times, maybe it's worth a blogpost on its own. It's mainly a reaction to recent posts of Dave West from InfoQ, and the discussions formed beneath them. Although I can feel it through what makes someone to think this way, yet I have different opinions which I would like to tell.

I have been hearing a lot of times about wether what programmers do is engineering, is science, art, or what. When I was about 18, even I had some thoughts that building software is not about engineering, but now, perhaps just because I became a master of software engineering officially (ok, it's called engineer-informatitian in hungarian), I do think it is.

Why does the question arise? Because our daily job - at least, for a lot of us - is not based on science, but rather is about some chaotic finding-your-way thing. It rarely involves drawings and science - not gut - based calculations if you don't explicitly insist on them, especially not in the enterprise world.

Some say 60 years ought to be enough for an engineering discipline to form, and therefore this isn't one; I think it otherwise. I think it will take us a lot more time to find out what this thing is, even if we reached this far, and even if our profession has roots in the ancient Egyptian civilization (have you ever thought of that the basis of Egyptian tax administration is a series of calculatiosn based on water sensors and other aggregated data?).

Let's start with two questions: what is engineering? What is software engineering?

Let me answer the first question with a personal point of view, and a second with an official one.

Archive: Testing legacy web applications using a black-box method

Cross-posted/archive:Originally a comment on an InfoQ article, later became an article on my personal blog

Recently, a whole module of a legacy web application written in PHP4 around 10 years ago (and constantly "maintained" since) was needed to have new features.

We're talking about 1000s of lines of code within one function, or even case of a switch case.

Most of the time, people don't refactor maintained legacy applications, as somebody told me "the first rule of support development is: don't change anything other than requested, just add your stuff."

I haven't been able to track the application back to its beginnings but its imminent to me that if-else branches don't grow to 600 lines by themselves, without human intervention. Somebody has to mess these up, and someone has to have such kind of thinking. This is pretty general in enterprise programmng:

• most of the tasks are about legacy applications


• people fear to clean things up


• it's not about development, but adding requested features and fixing bugs.

Also, PHP is a dynamic language, and therefore formal refactoring tools are usually unavailable. For example, PHP refactoring support in netbeans is basically non-existing.

So, what would you do here?

I decided that a system's answer is dependent only from its input and its context. This seems pretty straightforward:


System ( input, context ) -> output


OK, what is the input of a web application? Of course, its HTTP request! In PHP, it's hard to think about any other input.

What's the context? Context is given by two components basically: the underlying platform, whatever it is (no matter you have a framework or just common libraries, we call these platform together), and the persistent data layer. So:


Web app (request, persistent data) -> answer


(I know, I forgot platform to add, but in refactoring scenarios, platform should stay the same anyway. In case you change platforms too, there are other complications which we won't talk about this time.)

What's the answer? First, it's an HTML (or XML, JSON, etc) output. We didn't have to care about it in this particular case. The other output is: changes to the persistence layer. It's unusual for web applications to change anything other than their database and cache layers.So:


Web app(request, persistent data) -> (written-out response, persistent data' )


OK, what to do? We have an old system and we want to refactor it to a new system, and the question was: are they equal in functionality?

Question is: Web app == Web app' ?

Let's see what I did:

• Ask a manual tester to go through every possible combination on the user interface
  
• Recorded these into files (serialize($_REQUEST)), or, even better, (serialize($GLOBALS))
  
• Ask the DB layer to NOT write anything to the DB (ugly global variable hack, if it is present, only select queries are executed), this way, we ensure that we keep a consistet state
  
• Record every writing operation (so, instead of executing them, take note of them)
  
• An algorhythm:
  
  
  
  1. load the serialized request,
  
  
  2. start recording db,
  
  
  3. run the original controller,
  
  
  4. collect db recordings,
  
  
  5. re-load request (in case it was modified by the original controller - we could never know)
  
  
  6. run the new controller
  
  
  7. collect db recordings
  
  
  8. see if the two are equal
  
  
  
  

This way, I could be sure, that in all of the scenarios a manual tester could come up with, both of the controllers behave the same way.

After the original recordings, I did a few additional points:

9. re-load the request again
   
10. enable db writing
    
11. run the new controller
    
12. display result.
    

This way I could create an - albeit slower - but seemingly normally functioning version of the software, which did everything it did previously, and it was verified that functionality haven't changed with the new controller.

I called this blackbox-harness test.

What do you think?

What's this blog?

Back in 2009, I started a category on my personal blog, called "Architect Things". They covered some of the every-day problems I faced while working on a project. This is the continuation of that project

Who's this guy?

My name is Adam Nemeth. I'm a software engineer MSc. You can see my CV at LinkedIn. Usually I run under the name @aadaam (which is both my twitter and facebook account)

My main field of interest is applying software engineering theories in real-life situations. Normally, I'm at the frontend side (where the users are), and using dynamic languages, like JavaScript, PHP or Python, which makes my adventures a bit more weird.

Nonetheless, UML doesn't care about which language will you implement in, nor do real principles. In fact, it's a good test for them to check wether they're that universal :)

As you've probably guessed, in the last few years, I was doing software development teamleading, and large scale software design for large companies.

What could I expect here?

High-level thoughts on software engineering, including, but not limited to real-life modeling practices, architectural and design patterns, and some general ideas which are not directly about implementation. There are too many people speaking just about those...