Originally posted on the Futurice blog, this post offers a broad approach for developing software in an enterprise context. The ideas presented here are based on the methods on my own experiences and those we use in at Futurice so with some luck, you come away with some ideas that you can use to develop your own approach.

Developing enterprise-level software is very difficult and failures are extremely expensive, so reliable ways of working that really deliver results are vitally important. Standard practices are luckily changing for the better.

There’s a lot to manage in an enterprise-level software development project and challenges come from a number of different aspects:

• Extensive product requirements
• Many stakeholders
• Complex range of legacy technologies
• Mission critical, strategically important, or both
• Fast and flexible development within existing software management processes

All easier said than done.

The old-school approach

Traditionally large, complex software projects have been managed using structured methods based on analysis, design, coding and testing - in that order. Developers are typically organised in functional teams and they usually have little visibility the workings of the other teams they deal with. Integration is not continuous, Release cycles are long and iterations were slow. Needless to say, projects operating under in this way often run over budget and deliver late.

In the beginning of my career as a software engineer, this was the standard practice. A number of projects were delivered but it was not easy and the long feedback cycles often meant the significant mistakes were made whilst expensive redevelopment was common. Integration was a nightmare.

A leaner approach

My first exposure to lean principles was in 2002 whilst I worked in the production planning department of a German research institution. Lean manufacturing was hot and its application to the software domain was often by way of a set of principles collectively called Extreme Programming. Rightly or wrongly, in my mind this was the beginning of Agile methods in a software development context. Start from the point of view of the customer and deliver those features which lead to the customer’s satisfaction. The idea of Extreme Programming (XP) was that it improved a software project in five essential ways; communication, solution simplicity, continuous feedback, respect between customer and developer, and confidence to move forward with new features and development. However it was widely misunderstood and difficult to implement in most environments.

In stark contrast to the traditional waterfall approaches, XP methods called for continual user feedback and iterating functionality based on that feedback. Requirement specification documents without end became a thing of the past. In fact, Extreme Programming was incorrectly associated as a methodology which called for no documentation. It does in fact call for documentation but only for documentation that adds value.

Although the roots of its history stretches back to the 1980s, the mainstream emergence of Scrum as a methodology in the early to mid 2000s saw it as far more palatable to the corporate environment. It’s perceived structure was stronger and it’s documentation process was clearly defined. Corporates took note and soon enough many were calling their projects Agile - something that couldn’t be further from the truth. Although not perfect, the value of agile methods could be seen as a way to tackle the difficulties surrounding software development.

Over the years, we’ve developed a set of ways of working which borrow heavily from lean and agile methods. Without a doubt this has been a large contributor to our track record. In this article, I will attempt to explain the management methodology that we use to run software projects. The methods are simple however it is our discipline in execution that has made it so successful. In point form, this means:

• Co-creating ideas
• Organising around self-organising, end-to-end teams
• Integrating early
• Managing your backlog carefully
• Releasing software fast
• Validate continuously
• Using KPI’s to measure success

Co-Creation

This is where it all starts. Here we aim to tear down the functional silos and bring people together to create interdisciplinary teams. Thus supporting co-operating in a way that produces a mutually valuable customer outcome. The important thing here is getting people collaborating to focus on value and how to maximise it in terms of the project, the business and importantly the end customers. Product owners, designers, developers, quality assurance and analysts all in one room, sitting together, building. You can probably start to imagine how this facilitates communication between the project stakeholders. Validation starts from the get go and our experience shows that if you want efficiency and the best outcomes, this is the place to start.

In fact, co-creation encompases a broad variety of stakeholders. Business people use co-creation to identify value whilst working in a customer centric way, engineers use it to incorporate design thinking into their agile development methods, and experienced service designers use co-creation as a way to better understand the problem worth designing for. The reason co-creation is so powerful is because it brings together diverse opinions. Diversity of opinion is what truly drives stronger outcomes because it supports a far higher level of efficiency in the end product. When opinions align, as often happens inside functional silos, inefficiency is typical and outcomes are subsequently hindered.

In other words, the real power of co-creation lies in the fact that it brings multiple views of the world into the analysis process. When the stakes are high it is dangerous to project one’s own understanding of what is right and wrong onto the solution at hand. No matter how smart or how widely read, one views the world through a particular frame and we look to confirm those views in the information we seek and the solutions we produce. Co-creation is the simplest way to start working around this natural human bias so we use it extensively.

Self organising, end to end teams

Traditional development teams setup based on function. That is, Database administrators, front end developers, back end developers, network administrators are each organised into functional silos. Intuitively, this makes sense as it allows project members to specialise whilst project managers can optimise their teams in terms of costs. Unfortunately, this couldn’t be further from the truth. The most visible consequences of this are longer lead times and higher energy spent coordinating activities. Further exacerbating this is each team has very little visibility or understanding of how their outputs interact with the other aspects of the system. Waste becomes common as handoffs often lead to re-work, quality issues, bottlenecks and delays. It doesn’t take much insight to realise that all of these issues lead to increase costs - in contrast to what we are actually trying to optimise in this setup.

The developers who work with us vary in their experience but overall the common thread between them is that they are good at what they do. Rather than organise by function, we organise teams vertically so they are cross functional - a speed optimised setup. With this in mind, we provide our teams with clear constraints with which they can work within and it is then up to them to organise themselves within this boundary to choose the best way forward to get a project done. With that comes a lot of responsibility and trust but our experience over the past years has shown that in this setting people tend to deliver their best. We encourage teamwork and collaboration so that project members can play to their strengths. It’s a potent mix but some might consider this as a leap of faith - one we are happy to make. Give project members the room to achieve along with the resources they need and the chances are that the outcome will be exceptional.

In my experience, horizontally organised teams as described above (cost optimised) more often than not waste considerable time on integration between the layers. In fact, the experience is one I hope I won’t have to work in again. Vertically organised teams (speed optimised) on the other hand are rewarding work in and progress is rapid because integration happens as the software is developed. Integration almost feels like a non-issue. Can you imagine?

Integrate Early

System integration is often one of the most painful exercises a software developer can do. It doesn’t matter whether it’s integrating a front end with a backend or some legacy subsystem, integration usually involves more than just software development. It also involves strong communication and problem solving abilities. It’s best to make integration continuous - from both automated and human perspectives.

Our self organising, vertically organised teams start integrating from the get go. As explained we actively encourage cross functional teams that include design, front end developer and back end developers. This creates an environment that fosters strong communication and collective ownership of the entire system - a massive gain in terms of productivity, team member involvement and commitment. The subtlety here is that any member of the team can fix the problems they find - if they can. This is not about who owns what. This is about building the system in an environment where the lines of ownership are blurred. With this in mind, members are empowered to integrate “as they go”.

It’s not just the team setup that supports integration. We also make ample use of continuous integration (CI) tools such as Jenkins, Circle CI and others - depending on the choices of the team. Continuously integrating in an automated fashion is important because it catches problems early. As code is committed into the source code tree, a suite of tests are run against the code base and problems reported. Errors are found immediately and fixed quickly. With this in mind, developers can be sure that they are integrating, and thus minimising future problems, as early as possible. This shorter feedback cycle makes big sense.

Managing the Backlog

In the traditional waterfall model, deliverables are managed based on the scope and deadlines (milestones) of the project. Many things tend to be ongoing at any one time and deadlines inevitably lead to a final crunch time as cutoffs loom large. Multiple tasks are often thrown at developers and it is up to them to keep it under control. The unfortunate side effect of this situation is that team members become less productive, shortcuts are taken to get it “out the door” and ultimately a reduction in output quality is realised.

Another unintentional side effect of this mode of operation is an increase in complexity. With high levels of work in progress (WIP) the need to manage inter-dependencies and regression quickly emerges. Think about this, if we are working on many incomplete tasks it is only logical that it takes longer to achieve a complete working body of work. This means release cycles tend to be (much) longer than if we fully complete small bodies of work in a focused way (low WIP) and release as we go.

Maintaining a minimal backlog of WIP is a key point to understand because the cost of processing change is expensive.. In many projects and indeed in many aspects of life, the longer the list of things to do the more difficult it becomes to stay focused on the task at hand. It is too easy to context switch to a more “enjoyable” task. When we context switch, we lose time and the fidelity of thought. Cognitively we simply can’t seamlessly move from one task to another. Context switching must be minimised if optimal output is to be achieved.

Let me put it another way, no one can do more work than they can handle. Seeing this statement written down makes it seem obvious however in practice it is not so simple. Managing the backlog of stories and tasks being worked on is a critical skill for ensuring maximal development speed. Let’s get one thing straight, where possible we avoid parallel development that might see a team working on a large number of stories/tasks at any given time. It’s more than we (as humans) can handle. Thus, our goal with backlog management is to keep WIP to an absolute minimum.

With the aim of minimising context switching through lower levels of wip, new tasks can then only be pulled by developers based on the story and task priority and the ability to complete. The traditional system sees new work pushed onto the developer regardless of where they are. Our way is to pull tasks when the developers are ready and to maintain that flow of tasks moving through the system. It’s a subtle yet very important difference.

Release Fast

Returning again to our scope driven waterfall model that has traditionally occupied the software development space, traditional projects tend to be managed in order to deliver the scope of the project. Customer and user requirements are a secondary concern. Under this model, many times we have heard customers say “I know that’s what I asked for but that’s not what I meant”. To which developers reply, “You could’ve told me that!”. This common scenario sees both customers and developers left feeling unfulfilled.

With higher levels of WIP and their longer release cycles it’s no wonder the client isn’t getting what they wanted. Too much development goes into making a feature which makes it difficult and expensive to change later. Now we know better.

With levels of WIP minimised we can now start to deliver shorter release cycles. This is a nice side effect because it helps to provide customers more visibility of the project state. Product owners can actually see working features as they evolve. Moreover, this helps product owners prioritise the features they need now as the feedback loop is far more fluid and efficient.

However it’s not as simple as minimising WIP. The team setup and software infrastructure need to support continual this integration of new development into the existing code base. We’ve already covered our end to end team setup so we should again briefly discuss the role of automation and how it allows us to move quickly. In fact, this is one of the key components of our iterative “release fast” mantra.

To recap, old school projects in the scope driven world tend to develop code now and integrate later. This is a poor software development practice no matter which way you look at it. As mentioned earlier, in our way of working we make heavy use of continuous integration (CI) and continuous deployment (CD) tools. CI is of particular importance because it sees every incremental change committed to our main code base tested for functionality, integration and regression. This does not happen in the rush two weeks before “go-live”. It happens on every merge, every day. We hit problems early and we solve them quickly. Subsequently software development is done on working, well factored and tested code. Needless to say, the foundation from which new development can take place is far more solid.

If code is integrated, tested, well factored and client approved then what’s holding us back from releasing now. Nothing stops us - so we do it. And we do it often. In fact, because the client has been involved along the way we can start releasing and validating immediately. This incremental approach is an incredible gain over scope driven approaches.

Validation

By now, we have a solid framework from which we can start validating the fruits of our labour. We are co-creating, integrating early and releasing quickly however we still don’t know whether anyone actually needs our product. It goes without saying that products must be validated early on in the piece. That is, it is critical to seek feedback about the new product, in some form or another, from the target markets who we anticipate will use the product.

Many new businesses start with a comprehensive plan of what they are looking to achieve and how. Although this detailed planning provides comfort and makes project justification easier it is in large part a wasted exercise. New products are just too uncertain to make such detailed planning useful. Instead, starting from a broad project vision we use the principles explained in this article and we back them up with customer validation and adjust where necessary.

Although validation is no guarantee of success, it does reduce the likelihood of waste. Not only are we validating that we are on the right track, we are more importantly finding out whether we are on the wrong track. Consider how many products have been built at considerable expense in the absence of validation only to fail after launch because no one actually wanted the product. The number of projects fitting this bill is endless. Again, consider how much time, effort and money could have been saved if the product was validated early on in the development cycle. The product strategy could have been pivoted towards a new direction or even abandoned entirely. Validation is about introducing a customer focused feedback loop into the development lifecycle. Develop, validate, refine, repeat.

Measuring Success

Earlier it was mentioned that we are no longer measuring deliverables against scope and schedules. This extends to the measurement of project success. You might be surprised to read that delivery of the scope is no longer a measure of success and that we do not treat the velocity of user story completion as a KPI in our development - it makes no sense. Our experience is that this is counterproductive to quality software production because it perverts the incentives for well factored, well tested code that meets or exceeds the customer’s requirements. With a focus on velocity, humans naturally tend to take shortcuts to meet deadlines. Moreover, as managers, in this mode we tend to start making estimations and promises that we can’t meet as time starts to become the focal point of the daily routine. If you deliver a project on time and within budget but the software does not do what the business had envisaged then who has the project been a success for? (hint: it’s not the customer or the end user)

As many software developers have painfully learnt, software development timelines are often a waste of time and estimating the time it takes to deliver a complete piece of code for a feature can be difficult. It takes practice and an acute awareness of your project domain and even then developers will tend to be well off. With this in mind we take a different approach that we reason makes far more sense in terms of considering software as an integral part of a business strategy and operations.

We look to measure success against KPIs which reflect the value our software delivers to the business. For example, growth in sales, increases in margin, increased conversions, etc etc. As soon as our clients can let go of the traditional mindset of scope and delivery and move onto business value delivered, project sponsors can only then start to understand the value they are receiving. By focusing on KPIs, feature creep is also minimised. Does it add value and does it increase the KPIs? If not, it may be that the requested new feature will be an overproduction which will provide a limited increase in value.

We generally measure two things. Firstly we measure which features are of priority and we work on aim to deliver those first. Working closely with the product owner we are able to ensure that the things we deliver are the things the client needs now and which things the client attaches greatest importance to.

Secondly, we take actual business metrics to measure the success of our projects. For example, sales growth, margin expansion, conversion rates, etc. We could have the fastest project delivery ever but if we are not improving our business metrics then what are we really doing? This is how we wish to be measured. That is, measured by the business impact our work delivers. We are confident to stand by this.

The result of this is that we tend not to promise project delivery dates and for many organisations this will be difficult to understand. Reflect on this for a moment. We do not work to delivery dates. Instead, we work to deliver the features we can produce in the allocated time and budget.

There is another benefit of the KPI driven approach. A focus on priority driven deliverables focuses a client’s mind on what is important whilst allowing us to start focusing on the quality of our work over the quantity of our work. This means we focus more on well factored code that is easier to test and scale. The net result of which is a sustainable working pace, higher productivity, better quality code and ultimately more meaningful software.

Conclusion

I have presented a simple and modern approach to software production that can be used to achieve good results across projects. It is heavily customer driven, it is iterative and it is sustainable across both short and long running projects. Just as important, it encourages close collaboration between teams, project members and clients. In our experience this Agile way of working has been of great benefit to us and it is a large part of our success. For the readers of this article, the beauty of what has been presented here is that it is both repeatable and reproducible across a broad number of domains.