organisational issues

The Perfect Engineering Manager

October 2, 2019 by Greger Wedel

In my last post I introduced a practical approach to engineering leadership and the skills required in any engineering organisation.

The post must have struck a nerve as it got a lot of reads, especially on Medium in The Startup magazine where I cross-post some of my blog posts.

However, I’m sure people have many new questions on the practical side of how this works in practice. The key insight I tried to convey was that all engineering organisations are built around four distinctly different skill sets and that you need engineers who jointly cover all the skills, no matter how you distribute the responsibilities: code development skills (the developer), task priority and tradeoff handling (the backlog owner), technical architect skills managing long-term vs short-term (the architect), as well as people and team skills to keep people happy, make them work on the right things, and make sure things get delivered (the manager).

The larger the organisation, the more roles tend to be specialised. As a result, the coordination cost goes up. You typically then get elaborate frameworks and formal development processes to ensure coordination. This is a drain on time and motivation for most software engineers. On the other hand, in successful and small startups, you typically see very flat organisations where everybody seems to do everything. How then can you successfully scale an engineering organisation and keep the energy and productivity? Amazon’s 2-pizza teams is a famous approach, but reality is that it all starts with the people you recruit, so you need to start early. If you hire a project or program manager who has been managing projects and processes, but without delivering own content into these projects, you will get more projects that are managed by process, not by purpose.

And “purpose” is a keyword. You want people who are driven by purpose, not process. And here we get to the Perfect Engineering Manager. If you have one Perfect Engineering Manager on each team, focus will be on purpose and you can reduce coordination to a minimum. So, who is this Perfect Engineering Manager? In fact, he or she probably does not exist, or there are very few of them out there. Luckily, the skills can be trained. Not surprisingly given the lead-in to this post, the Perfect Engineering Manager is a person who is a brilliant coder, loves enabling and helping out others to make them successful, takes natural responsibility for delivering high-quality code into production, knows the code base inside-out and will re-factor quickly while adding new abstractions and code that will help others deliver in the future. Mentoring more junior developers comes as a given, and the natural authority of the Perfect Engineering Manager makes it a breeze to drive prioritisation, sprint planning, daily stand-ups, or any other chosen development approach. Of course, the social skill set of the Perfect Engineering Manager means that other developers love working for her or him and necessary interactions with other teams are always pleasant and smooth!

The Perfect Engineering Manager is an ideal and getting closer should be an aspiration for well-rounded, senior engineers. Not in the sense that everybody should strive to become Perfect Engineering Managers, but that one or more skills can be chosen and improved as part of developing as an engineer and a human being.

Without at least one such person, or a team of people who together cover the Perfect Engineering Manager skills, you will not have a 2-pizza team, you will have a small group of developers that rely on others to get things done. Developing great software and great products is not rocket science, but it requires systematic and careful grooming, development, and attention. In too many organisations, software engineers are left on their own without much help or guidance on how to grow their career, job satisfaction, contributions, and as human beings. The Perfect Engineering Manager is of course instrumental in facilitating and making the right decisions to make sure that the engineers on his or her team are not in that unhappy group.

A Practical Approach to Engineering Leadership

August 17, 2019 by Greger Wedel

In my previous post on the new engineering leadership skills needed in a devops world, I called for a more systematic approach to developing leadership capabilities in engineering. But I gave no answers. In this post, I’ll share my practical approach to developing leadership capabilities in an engineering organisation.

When developing great software, there are some fundamental questions all engineers have:

What can I decide?
Who can make decisions on things I cannot decide?
Who do I need to talk to or involve when resolving an issue?
Who determines the targets for what we make, how we make it, and how we go about doing it?
What is expected of me and what kind of rewards do I get when I meet expectations?

In a small startup team, you don’t really need a lot of these things explicit. The goals, actions, and rewards are mostly shared across the team, and everybody contributes where they see a need. The entire team has the same priorities, the tradeoffs are decided upon continuously, and changing course is done quickly.

However, you cannot grow much as an organisation without introducing slightly different priorities in different teams or roles. You are starting to introduce the cost of coordination. For a while, you can align everybody around the same priorities through some kind of coordination process or meeting that gives you the “top 5 things to work on this week”. But to allow rapid growth you need to at some point allow different people or teams to focus on different priorities. I have seen numerous approaches to solving the resulting coordination problem, and most either introduce too much process or too little.

There is no single answer that works for everybody in every situation. However, if we go to the fundamentals of how to build great software, there are four different areas that always have to be covered in some way:

Build high quality code, designs, and other tangible stuff
Knowing what to make to solve the customer’s problem
Knowing how to technically solve the problems and deliver something that will work over time
Knowing how to go about making the tangible stuff like code and actually delivered what’s agreed upon

The skills that go into each of these four areas are very different. And the more people involved, the more coordination and collaboration is needed to make sure everybody is pulling in the same direction. Ideally, you want everybody to be 100% productive, but that is not possible as long as there is some kind of dependency between what people are supposed to do. How do you get the productivity percentage as high as possible?

Typically, each of these responsibilities translates into one of four roles (not individuals):

Developers build stuff
Backlog owners manage the priorities and the definitions of what is to be made
Technical architects determine how to technically solve problems
Engineering managers lead people, the process of development, assign the right person to the right task, and ensure velocity

These are roles, not individuals, so many engineers will wear multiple hats. Also, there is one missing part here: the service or product. Each organisation has to break down its services and products (and thus code bases) and group them in a sensible way. Once that is done, it should be possible to identify the developers, backlog owner, the manager, and the architect for each. An engineer can be a developer on one code base/service, an architect on another, as well as manage a deliverable.

This break down into modules and components is absolutely critical to the productivity of your engineering organisation and subject to Conway’s law.

Unless you match the identified ownerships and roles with your modules and components and get that right relative to your organisational structure, you are introducing inefficiencies in your software architecture and how people communicate. This is a continuous process and needs to be adapted to the changing needs with respect to both features and the underlying system architecture.

Going back to the questions at the top of this post (that most people have in a software development organisation), it should now be possible to answer the questions. You don’t have to go through a central manager or (..!) a committee or a process, you can reach out directly to the person in charge who can make a decision. E.g. if you are a backlog owner and you need more engineers to work on your backlog, you can talk to a manager. If you need an evaluation for how to implement a certain functionality, you talk to the architect. If several components need to communicate through an API, you know exactly whom to invite to discuss this.

You thus get an organisation that can resolve most day to day matters without central coordination.

Typically, you will have triplets of lead people for each of your main components: a manager, an architect, and a backlog owner. The development team can thus be productive by getting help from each of these three. Ideally, all three roles/people are co-located physically for highest productivity since high velocity requires quick and frequent touch points.

There are two other roles missing in this picture: if the team is a product team, you need a product manager role (who could also be the backlog owner, dependent on skill set). If your component or product has a UI, you should also have a UX lead who can work with the backlog owner and product manager to get clarity on the deliverables. See also an earlier post on the backlog and the product owner role.

This model is also good for training and career development. There are four areas where skills can be built dependent on individual interest. As engineers grow from juniors to become more senior, they will take on more roles and expand what they take ownership of. Some may want to take on manager roles and lead teams. Some may want to deep dive in a very narrow technical area and become lead architect.

Remember that any changes in an organisation will take time. You need to repeatedly clarify the expectations, the opportunities, show that real authority to make decisions come with these roles, and systematically encourage people to take ownership. And if people are uncertain, you need to exemplify what concrete tasks and actions are necessary and expected.

The reward is an empowered engineering organisation that can deliver great software with high velocity!

The Program Manager in the Agile/DevOps Organisation

March 24, 2016 by Greger Wedel

This is another post in my series on transitioning from waterfall to agile and devops processes. This post covers the program manager role, a rather unassuming role in some organisations, an important role in others, but nevertheless a role where the right person can potentially make a big impact. There are a few different types of program managers, so let me be clear on the type I will discuss here. Microsoft uses the title for product managers with a horizontal responsibility, i.e. functionality that should go into multiple products. Other organisations use the title for a role that is similar to a product manager tole. Typically, a program manager of this type has an independent responsibility for business results within a certain area. A very different program manager role is a project manager for a deliverable that crosses multiple independent products or teams. Often, a program manager of this type will work with multiple project managers, one for each product involved in the deliverable, but does not have an independent ownership of business results and thus becomes more like an advanced project manager. In Cisco, program managers are often of this last type (and clearly lacks the first type), and alas, some larger products may have their own program manager working alongside the project manager to manage and coordinate outside the product team.

I may cover the consequences for organisations not having the “program manager with business ownership” role in a later post, but here I will cover the “program manager as an advanced project manager” role.

Of the many roles in a development organisation, the program manager role may be the role that involves the most painful change in mindset and daily tasks when a large organisation moves to agile and devops. The traditional tools of a program manager is no longer of much use: defining deliverables, setting and tracking milestones, managing dependencies, measuring progress against the plan, reporting status and escalating issues to stakeholders… But why is that? The reason is that the traditional program management model and associated tools assume a planning horizon (e.g. ship within 6 months), a clear view of what is “inside” and “outside” of the boundaries of what is managed (e.g. we need to ship a and b and rely on c and d to deliver x and y for us to be successful), a desire and need for a pre-deterministic plan for what is “inside” (e.g. we are going to ship a,b,c, and d), and that any deviation from the plan, including “outside” changes, are bad and should trigger a corrective action or some kind of mitigation.

ALL of this is wrong for agile and devops! For one, changing plans is desired, that is called learning. Also, the boundaries of what is “inside” and “outside” may be shifting as the teams learns and gets feedback, and reacting to change is something that each member of the team should do immediately. Planning should be done iteratively based on learnings and changing requirements, and what is actually shipped is not fixed as teams commit to a prioritised set of tasks (backlog), not a fixed set of deliverables.

Alright, so the program manager role is dead, is that what you say?! Yes, the OLD program manager role is dead, really dead. However, the reasons for why the program manager was needed are still there: ensure that multiple teams and projects with independent execution paths coordinate priorities and execute efficiently so that real cross-product value can be delivered to the customers with the right quality!

Can program managers re-invent themselves? I really don’t know, some of them may be stuck in old ways of doing things. There are of course many flavours of agile that companies may choose to adopt, like SAFe, where you get clear definitions (http://www.scaledagileframework.com/rte/) of what a program manager is supposed to do. The problem I have with such frameworks and role definitions is that activities, tasks, and goals are often defined in terms of artefacts and ownership of activities defined by the framework. Thus, the role quickly becomes statically defined, which is contrary to the whole point of agile: constant improvements reacting to learnings. By the way, as a rant on the side: that is probably also the biggest problem with frameworks like SAFe in general. They capture some important learnings and concepts, but become stale and not a framework for change.

I really like to get to the core of what we are doing and what we are trying to achieve. E.g. “Manage risks and dependencies” is an abstract statement, and it involves too many implicit definitions of what a risk is, how you manage it etc. But looking at the problems I have seen in agile and devops, a typical program manager personality can contribute buckets to some of them (but constantly change HOW to contribute to them as the organisation evolves):

1. What do we want to deliver and when? In large programs, multiple product managers need to agree to a set of ordered business priorities. Without clear and shared priorities across the products that are to be delivered together, we get absolutely nowhere. This is the role of the product managers (or rather ironically the role of the other type of program manager in organisations that have those). As covered in the post on the backlog, I believe in a true and single owner of a backlog (delivering on the priorities), and the backlog being defined as the task list a development team uses in their iterations. When a team of product managers come together to define shared business priorities, the team of product owners need to come together to tie their backlogs together (as described in my backlog post). A group of peers with priorities outside the group has always problems with closing decisions because conflicting product-specific priorities will interfere. A program-level product manager (aka solution product manager aka that other type of program manager.. puh!) can be used to drive the program/solution priorities and resolve the conflicts between the program-level and product-level priorities. Likewise, a program-level product owner can work with the product-level product owners to tie the backlog items together so that co-development can be done (where co-development means actual coding done in the same iteration with developers talking to each other). This program product owner role can be filled by a program manager (assuming some technical knowledge).

Solution Product Manager ——— Solution Product Owner ——— Solution Program Manager

|| || ||

Product Managers Product Owners Project Managers

The flow is thus: define and prioritise shared business priorities —> define interlinked product-level backlog items to deliver on business priorities —> plan co-development where needed. I tend to talk about “backlog items that are flagged for co-development”, which basically means that there will be another (named) developer in another team (or more) who should be involved when writing and testing the code for that backlog item.

<Whining rant> When writing this, I really feel the urge to go back to a startup environment and the simplicity of making decisions and pushing things out the door… The less people who don’t code, the higher feature velocity you get. But then again, you typically have just one or a few products and limited coordinations needs… </Whining rant>

Another important thing in devops is metrics and A/B testing. This input should be continuously brought into the business priority process. Often there is a breakdown in communication between engineering (doing metrics and A/B testing) and product management (setting the priorities). I have yet to see program managers really making a difference here, strong architects and engineering leads tend to basically take ownership of parts of product/service improvement, but I believe this is an area of real value creation where a good program manager can make a difference.

Another important aspect of program-level work is to set the right targets and make sure engineers can focus on getting things done. It is important to go to the right level of detail on the program-level of what a specific program backlog item really means. Getting to the right level of detail on wireframes, prototypes, and requirements through demos, transfer of knowledge sessions, and white-boarding is important to make sure that the engineers from all of the teams involved in co-development have a shared understanding of what needs to be done. If there is one person responsible for driving this on behalf of the product owners, things seem to move faster.

Of course, another important part of this phase is to get the architecture right, engaging the senior engineers in each of the teams and preferably with a senior architect with program-level knowledge and ownership to lead the process. In Cisco, there is a separate technical leadership career path where director-level technical leaders (principal engineers) are expected to define and supervise architecture in larger programs and also across programs and business units (I’m btw one of those).

2. How do we tie together iterations to do co-development? Once the priorities have been agreed to on the program level, the engineering managers and product owners need to coordinate and identify when and how to do any needed co-development. This will include backlog grooming, developers planning an iteration together, sharing builds, and coding together possibly on a day to day basis. Two and two teams can manage this nicely on their own, but if three or more product owners and engineering groups need to coordinate, there is often the need for a facilitator. Again, this is more of a solution-level product owner type of role, not a program manager, but a technical program manager would fit nicely in this role.

3. How do we bring together the products, validate that iterations delivered planned outcomes, and qualify the process towards release/deployment? Once an iteration has been done, there is a need to bring builds of each of the products (or services) together to do testing and validation of the deliverable on a solution-level. In a true devops environment with continuous deployment, you will get this validation quickly, but if you deliver products and not devops services, you will need some solution-level live test environment. Solution-level testing needs to be done (again, preferably done as part of the iteration…), and if everything is ok, the builds can be deployed to a wider audience, either internally or to early trial customers. This is a critical stage as solution-level bugs will show up. There is a need for a strong person managing the maturity of features delivered and determining whether to allow the features to go beyond the initial solution test bed. In a devops approach, you will have developed features with #debug flags and feature-toggles, and features can be turned off if they are deemed not mature enough for deployment to a wider audience. The program manager can be an excellent watchdog in making sure that the right quality is achieved before releasing features to a wider audience.

4. Once a development cycle reaches an end and we want to ship, how to get things out the door? We want engineering to have a rhythm, a two-week pace that will give us some small, new improvements and features for every iteration. Ideally, engineering will not miss a beat when the products are released. However, more likely there will be a need for one or two hardening iterations at the end of a development cycle to clean up bugs, improve stability, and fix remaining issues. Once this has been done, all the code repositories should get a release branch, and the bulk of the team should pick up new feature development for the next release, while some time should be allocated to fixing bugs on the release branches and merging back to trunk. Dependent on the products/services, this final step before releasing products to customers can be long or short, but preferably it should be as short as possible. Typically, tasks will be final stress testing, solution-level manual testing, documentation finalisation, knowledge transfers to customer service people and sales engineers, release notes, bug scrubbing, as well as other steps to ship the software. This whole process play more to the traditional strengths of a program manager who is an obvious candidate to own this step of the process.

As you can see, with references to artefacts like backlog and list of business outcomes and to various roles, I get dangerously close to defining a framework of my own 🙂 The difference is that I here describe a(n ideal?!) system of people and deliverables that MAY work, but that must be adapted to people’s experiences, preferences, and beliefs through an agile process where people bounce off each other and change together. If not, the deficiencies inherently found in any human activity will settle and some important things will continue to be dropped, people will have overlapping responsibilities, and teams will work in slightly conflicting directions without the continuous improvement that agile promises.

But in sum, this post is an argument against the opinion that agile and devops mean that you don’t have to do any planning, documentation, or process management. On the contrary, some of your planning and process management need to be even more diligent than in traditional waterfall. So, don’t sack your program managers before you see how they can adapt…!

The Changing Development Organisation: The Software Architect

July 20, 2015 by Greger Wedel

(intro to this post on devops)

It has been a busy few months. Since my last post I have transitioned to a new role as part of day to day development (see linkedin.com/in/greger for more details) in the Collaboration Infrastructure Technology Group (UC, telephony, telepresence, conferencing, etc) in Cisco. Earlier, as part of Office of the CTO, I influenced the medium to long-term technology and architectural strategy, but as I had limited involvement in the day to day operations, it was a challenge to develop architecture that bridged near-term goals with longer-term strategy. Now, I’m less involved in long-term technology strategy, but on a daily and weekly basis involved in designs, decisions, and priorities made as part of development. Quite refreshing and comes with a bunch of interesting challenges!

If you have followed my blog, you have probably understood that I see architecture as something tightly embedded in day to day development, but more than just the sum of tactical, short-term steps. Also, you have probably detected my desire to see product management team and architects partner and work on the same thing: identify the whats and hows of building great products and services. Agile has for a long time changed my organisation and the last year or so, devops processes have really started to impact how we work. We have hardware projects, traditional waterfall product releases, and cloud services. This means that in my technology group we are currently using various flavours of waterfall, agile, and devops processes in parallel. It puts a strain on coordination and especially challenges existing roles in the teams. This post is about some of these changes.

Classic product organisations used more or less waterfall processes, and many so-called agile development organisations are using agile terminology to execute on a mostly waterfall mindset. However, with an increased focus on devops processes, the combined product/service organisation is facing another set of challenges in combining everything from hardware products and their software loads through legacy, monolithic, hosted applications to devops microservices. Tying these together to deliver robust, scalable services that offer great user experiences is a challenge along many dimensions: culture, people, competence, programming languages, tools, project management, goal setting, measurements, management, and roles.

One particular challenge is that not only does what we do change, our roles have to change as well. This is easy in a small organisation, but in a big organisation, the expectations other people have to what you are supposed to do is very important to your ability to get things done. So, let’s say I’m used to architects writing specs, then I will likely continue to ask for specs. Other people may not think an architect should write specs anymore, and they will ask for other things and maybe ignore the specs. This creates frustration and tension.

I believe most people in the industry are starting to understand the fundamentals of how devops are changing the roles of the development team and the operations team, and that they basically need to become one integrated team (with some new roles and new definitions of existing roles). However, the role of the product manager is also changing in the process, as is the project manager’s role, the engineering director’s, as well as marketing, sales, support, and partners. They no longer have releases to plan around and need to change how they work. I may cover some of these other roles in later posts, but in this post I will concentrate on the architect’s role.

First of all, a few words on what an architect is. In my understanding, an architect is a developer, a member of the engineering organisation, with a special responsibility for understanding the short-term and long-term business and technical requirements and making sure that the right technical deliverable is delivered at the right time with the right quality. Sometimes this is to get it done and deliver something less ideal from a code point of view. Sometimes this is to tell the product owner that some labour-intensive refactoring needs to be done in order to deliver a feature. My personal view is that a good architect also truly understands the value of the product to the end users and customers, and that he or she is a partner to the product manager or owner in defining and evolving the product. An architect can be a senior member of a scrum team (in Cisco, these are called technical leaders), or a more dedicated role that covers multiple scrum teams (senior technical leaders or principal engineers). In larger organisations, system architects typically span multiple products or services to create functioning solutions. In my view, regardless of scope, to be effective, the architects should be embedded in the engineering organisation. Some architects lead development teams, others have a more standalone role. However, the most effective teams seem to always three roles covered when decisions are made: the product owner (representing the vision and the problems to be solved by the product), the architect (representing how to deliver with the technical integrity of the product in mind), and the development manager (representing the developers, and making the trade-off decisions).

The architect is thus a member of the product leadership team. And with agile and devops, this role becomes even more challenging with the need to engage in more areas. Here are the some of the main areas of challenges:

Software architecture and code structure: There are two big challenges I want to mention here. First of all, devops requires a modularisation into micro-services, and getting the right grouping of functionality and defining clean, scalable APIs are critical. Second, the whole thinking of how you develop a feature or a set of features need to change. Every time you commit code, you cannot break anything. This influences how you architect the feature, how you test it, and you need to think in terms of A/B testing, feature toggles, and metrics, metrics, metrics(!), and not in terms of feature branches and unit and integration testing.
Software development processes: An architect needs to engage in code repository management, branching strategy, build ande delivery pipeline, test automation, A/B testing, feature toggles, 360 reviews, and much, much more. The architect needs to police the process and make sure that scrum teams working on various micro-services have the same view on the process and don’t break the builds. Very few other people will have the code knowledge and competence required to identify where the issues are and address them. The architect becomes critical to making sure that the devops machinery keeps working.
Requirements and product management: A shift from waterfall to devops is a shift in power from product management to the engineering organisation. Where product management used to ask for features based on market knowledge and research, a high speed of introducing features, doing A/B testing, and measuring the impact and quality relegates all product managers (except those who are very technical) to focusing on the go to market aspects of the service, not the requirements. The architect’s role in this becomes critical as the architects are instrumental in defining what to measure, and what you measure, you improve (whether this power shift has to happen, is a separate topic…)
Project management: Traditional project (or program) management should totally disappear. The software development process, the build pipeline, the metrics, the scrum standups, the scrum of scrum meetings, kanban boards, and the definition and usage of the APIs across the services, together with the automated tests together define and drive the project. The traditional tools of a project manager, like Gantt charts, dependency tracking, weekly project status meetings, read-outs to executives, escalations, and decision meetings based on analysis, are of absolutely no use in a properly executed devops process (note that badly executed devops processes will give ample room for such old tools and processes…). The architect needs to engage in the overall execution of the devops project to ensure that there is a constant evolvement of proper architectural frameworks, guidelines, automation, gates, micro-service designs, API/protocol designs, and much more.
Architecture: Devops does not mean architecture work is not necessary. On the contrary, as the architecture is constantly and organically evolving, the architecture that emerges can either be a disaster or a robust construction. However, the “ivory tower” architect is history as the emerging architecture emerges both with and without a plan, and it can only be shaped by having constant touch with what is being developed. A spec or design document will be obsolete within a week.
Management: I see a lot of very competent, senior architects who are brilliant at designing a flow or a new protocol, but who are lacking in managerial skills. Running efficient meetings with action items, pro et contra evaluations, closing decisions, setting deadlines and targets, chasing deliverables, and driving consensus are just examples of typical managerial skills that becomes more important for an architect.

There is so much more to how devops is changing an architect’s work, as always, the devil is in the details, but these categories have been helpful for me in understanding some of the challenges in evolving architects towards “devops ready.” The shift is similar already with agile development processes, but becomes more visible with devops.

Do you believe there are other important things that impact the architect’s role in devops? I’m happy to hear from you!

My next post covers the importance of one shared backlog backlog with all the actual work items visible, as well as the product owner’s role. Not directly focused on devops only, but relevant to devops as a good backlog grooming process is essential to getting anywhere.

Let’s forget about architecture and just get things done!

September 6, 2014 by Greger Wedel

In my previous posts on architectural debt, I focused on the engineering side. This post focuses more on the management side, or the organisational “gearing” factors that prevent systematic technical debt reduction. You don’t have to know what distributed system architecture debt is to get any value out of this post, but if you want to know more, read this post first.

When you reduce architectural debt, you are not doing a simple down payment with a currency like dollars. You need to do the right thing technically given your future plans. If not, you add to the debt, not reduce it. Or, you can also end up wasting time on reducing debt that you could live with. So, you need three things to do debt reduction: a target for where you need to go (based on your business objectives), a plan for how to get there, and the ability to stay the course, while also adjusting both target and plan as you go.

In my experience, there is no lack of good ideas for where you need to go architecturally. This can be in the form of short-term refactoring, an architectural model (borrowed or built), or longer-term standardisation work. The key problem is that the fundamental nature of distributed system architecture debt is that you can rarely fix anything through a quick-fix. You need a plan for how to evolve the architecture over multiple iterations, releases, and possibly over years. The more steps you need to execute, the more resilient to change you have to make the target architecture and the plan, because things change all the time, you have to adjust to achieve what you set out to do, and you will thus never end up where you thought you would be.

Let’s have a look at the typical steps you go through:

Identify key business objectives
Set long-term architectural targets that allow execution towards objectives
Make a stepwise plan for how to get to the targets (both transitioning for old products and a prioritised backlog for new)
Operationalise the plan with details on step 1 of the plan, and do feasibility validation of step 2, maybe also step 3
Execute step 1
Adjust target and plan for changes by starting at the top of this list…

You can choose your methodology, agile or waterfall, and you can choose your time line, short-term or long-term, but the steps are typically the same, but they are typically more or less formal dependent on your approach.

So, what can possibly go wrong?

1. Lack of clear business objectives and priorities, and architectural decisions cannot be made.

Often, management and product management will be able to define the high-level business objectives. But once these have been set, there are thousands of decisions that shape the overall architecture target (and thus the end product or service). Start-ups have typically been getting better at honing in on one set of customers and solve their problems (customer development, product/market fit, and lean startup are terms you may be familiar with). However, in a big company, there is a tendency to want to do too much in step 1. There is an inability to get to an agreement on a shared priority and say: “hey, forget about those big customers now, in step 1 we only want to solve the problem for these 20% of our customers, and then we focus on the next 40% of the customers in step 2.”

I often refer to the people doing this systematic planning and operationalisation of business objectives as the “translation layer”. Top-level managers lack the detailed knowledge, while the product managers and product architects typically lack the overall picture as they focus on their part of the business. The directors and technical leaders need to take ownership of the overall picture and establish clear priorities and objectives that both fit into the overall business objectives, but that also make sense for a specific product. Weak leadership in this “translation layer” leads to lack of guidance and constant adjustments in-flight when people discover that something is going in the wrong direction. You get the feeling of a ship with constant course adjustments, it looses speed, and if it gets bad enough, it will not have enough speed to even steer anymore.

Very often, the organisation fails to identify what I call the “pivotal assumptions” and make explicit decisions on these. When translation does not happen, people need to make their own assumptions in order to prioritise. Different teams will have different assumptions, and often the assumptions are not compatible. By identifying these underlying, pivotal assumptions and make the assumptions explicit and shared, the organisation will both execute in one direction, as well as be able to quickly change course if the assumption is no longer true and a different assumption must be made.

2. Lack of a strong architectural framework and principles.

We want each product manager, architect, and engineer to be able to make local decisions. In order to do that, they need an architectural framework where some of key technology choices have already been made and where a set of very explicit principles guide the decisions. Of course, without operationalisation of business objectives, it is difficult to establish a clear framework and principles, however, technical leaders have a responsibility to work with the commercial side to do this operationalisation. In this process, trade-offs are crucial. Foundational architecture work takes time and resources, and you don’t build the architecture before you deliver features, you typically need to both build architecture and feature at the same time or with as little time lag as possible. Thus, the architectural foundation needs to be prioritised in a such way that the desired features can be delivered in step 1. Also, when doing agile, you typically start out with a fairly loosely defined framework and principles, and then you evolve them.

When making these architectural priorities, you open up or close the ability to deliver certain features, and the trade-offs in what you will get needs to be put in front of company leadership. (I have worked on a light-weight approach to this that I have called “rapid priority evaluation”. I might blog about that in another post.)

I have found that in a large organisation, a typical hindrance to architectural frameworks and principles is that more than one team deliver products or services within the same architectural function. There is thus no clear accountability and ownership of the type: “you own that particular customer problem or technical function, go run with it”. I have championed establishment of internal “centres of excellence” for specific architectural functions. In a organisation doing acquisitions, this is particularly important because you loose a lot of development speed (and add to architectural debt) by allowing multiple teams to own the same function. How to organise a centre of excellence and what authority to give the teams could probably be a good topic in another post.

3. Focus on the target, not the step-wise plan.

Engineers have a prioritised backlog. The order of what you code first is extremely important to the end result. Especially if you have many teams, e.g. use scrum of scrums, and they are distributed across time zone, and there are dependencies between their deliverables. In order to get a good result, also business leaders have to live and breath the backlog. Too often, though, business leaders focus on the target, and forget about the management of the backlog. If a leader orders some juicy spareribs in a restaurant, the spareribs will be bought, marinated over maybe days, put on the barbecue 3-4 hours before serving time, and a similar “backlog” of work items is created for the rest of the meal. If the business leader two hours before the meal declares that he wants fish as a main course, everything breaks down and most likely the meal has to be postponed. An engineering organisation will start hedging when their leadership team over and over again change their mind at the wrong time . They will buy both spareribs and fish, do foundational preparation for both, and waste a lot of resources just in case the leaders change their mind. Engineering and architecture is exactly the same thing. But for distributed architectural debt, debt reduction is not comparable to buying the spareribs, it’s comparable to buying spareribs for a whole chain of restaurants! Imagine the impact when suddenly leadership wants a fish restaurant, not a steakhouse…

4. Inability to plan beyond horizon 1.

When focusing on targets, and not the backlog, leaders will “collapse” the organisation’s planning model from a multi-step, iterative process into a single target: what do we sell in 9 months!?

This focus on horizon 1 is typically supported in larger organisations by internal funding and project approval processes where planning is focused on projects with time-limited scope as opposed to funding of teams with responsibilities. In an organisation like this, people will have limited time and focus on what is beyond the next 9-12 months, and it gets very difficult to create that multi-step plan that will get you to a certain architectural target or reduce your architectural debt.

This lack of activities and formal planning beyond horizon 1 is a vicious cycle. You don’t get time to validate the feasibility of step 2 and 3 of your plan (or worse, you don’t have a step 2 or 3), bad decisions will be made and work will have to be re-done in the next phase. Also, very often you are not able to finish up what you planned in one go, so you will need to finish up in the next. Well, if you didn’t plan and communicate that need, you disappoint customers and you may not get the allocated time in the next phase to finalise. The typical management response is to tighten the deadlines, cut features, and manage the horizon 1 project “better”, maybe add a program manager to drive through the project.

5. Lack of transparent processes when adjusting course.

In our industry there is constant change. So, any plan you make will be incomplete and wrong. But, this doesn’t mean that a plan is not necessary. A clearly communicated and understood plan is a wonderful tool to drive a transparent process for making change. This is regardless of the size of the team or the project. If everybody understands what we are supposed do, we can easily discuss the implications of a change and make a more informed decision on how to make a change. And equally important, once we decide, everybody understands the impact of the decision, and execution will be faster and less error-prone.

Some engineering organisations have two separate career tracks (including Cisco): a management track (you become a director, senior director, and then vice president) and a technical track (you become a technical leader, principal engineer, and distinguished engineer). This is great for engineers as they are not forced into management roles to get career advancement. However, in order to track progress in a technical career, you need to accomplish something as an individual contributor. So, if a technical leadership career is about technology and individual contributions, a management career is about managing teams, projects, and making decisions. The problem is that while the non-technical processes and decisions are carefully managed by MBAs and engineers turned commercial, the very technical decisions are often made by a bunch of individuals who are all individual contributors plus some managers who don’t understand the implications. The technical leaders who are competent on the technical side of the decision do not drive a process, they focus on contributing their knowledge and facts, so that managers can make decisions. However, the managers who are good at driving processes and getting to a decision are uncomfortable and do not have enough detailed knowledge to make a decision. Then there are some technical leaders who are both competent and have the authority to make a decision and they do. This works great if just a decision was needed, but is less successful if what you needed was involvement from many stakeholders, evaluation of pros and cons, and bringing people with you to a conclusion that they are comfortable with and can execute on.

6. Inability to stay the course.

Building great products and services take time. Building great teams take time. If key people in the organisation either lack the confidence in the strategy, the ability to execute, or whether key decisions where right, it is difficult to get the commitment and funding to stay the course when step 1 is done. Managers with lack of technical insight mistake the need to adjust the course as a reason to change the course every time one phase has been delivered. In the restaurant metaphor, a single meal the first night is comparable to the first phase. However, a great restaurant needs to build up relationships with suppliers, develop their menus and recipes, hire chefs that are good at a certain type of food and so on. That is how you excel. Same thing for products and services. You are rarely done when you have completed a feature. You refine, you iterate, you listen to your customers and users, you track and you measure.

Too often, when phase 2 is next, what was done in phase 1 is forgotten, even the things we had to postpone because we never got to them. And if this is true for a feature, what then for steps needed to reduce architectural debt? When somebody makes a customer promise, or a competitor comes out with a new feature, management jump up and re-prioritise. The more they do that, the less debt is reduced, more debt is added, and feature velocity goes down.

7. Inability to recognise commonalities and patterns and a desire to start from scratch.

In our industry, we are always transitioning. Transitioning from an old product to a new, from an old protocol to a new one, from an old licensing model to a new, or from buying products to buying services. There is this rule: engineers always underestimate the time and resources needed to build something existing in a new way, and they love the opportunity to start with a clean slate. Truth is that the messy code we did to fix thousands of bugs and implement tiny features embody the accumulated experience of many developers solving a real-life problem delivered in a certain very specific way. If you want to solve the same problem, but in a very different way, you could start from scratch (though it would be wise to use the same engineers, possibly with some fresh blood). However, if you are transitioning your customers from one product or service to another and they expect continuity, you need to bring them with you. If you do that by coding from scratch, it is a very painful process.

However, by isolating functionality, modularisation, and layering, you can isolate pieces of code, leave some of it alone, and then replace functionality piece by piece. A carefully planned transitioning is beautiful to watch (but rarely gets much attention). The failures though get far more attention. A success requires a well-functioning organisation with patience and ability to stay the course.

8. Inability to execute well.

In larger projects with many teams and products/services involved and where you throw in multiple time zones and languages, the actual execution and coordination of your phase 1 may also fail. Lack of clarity and thus multiple interpretations across engineers, limited understanding of overall direction, too late and too little integration testing, limited availability of lead engineers and product owners who are experienced in cross-site development, and a number of other factors can lead to poor execution. This reduces the confidence in the ability of the organisation to deliver, increases frustration, and increases the possibility of the feeling of failure. The patience and room for doing proper planning and adjustment is reduced, and we go further into a downward spiral.

In sum, these organisational “gearing factors” magnify the impact of distributed system architectural debt. These factors are important in any software or hardware project, not only when you are reducing architectural debt, but the larger the organisation, the more projects, the more magnification you get. You have here the potential of substantial value destruction. But there is also substantial room for value creation if you use architectural debt reduction not only as a way to optimise your architecture, but also as a tool to transition your customers from today’s solutions and services towards your next generation of solutions and services. Or you can just forget about architecture and try to get things done…