organisational issues

The Three Deadly Sins of Scale-Ups

February 22, 2020 by Greger Wedel

So, you managed to pull it through. You found your product-market fit (or at least partially), got customers to pay, and secured your funding to expand. Your tight-knit team where everybody knows what everybody is doing will now double and triple in size as you add more people to the team to support your growth. Congratulations! Make sure you take time to celebrate because you are about to embark on a new and equally challenging journey!

The people you hired for the first phase were most likely generalists who thrive when things are unclear and problems just need to be solved without any process or long decision processes. You now need to decide who to hire for the next phase. Do you hire more generalists? Smart people who are individualists who will figure it out or do you hire people from larger organisations who can help you put things into process and scale up? Chances are that you are already establishing an HR organisation, and you are about to distribute the control of who you hire to those you have just recently hired. Most likely, the new HR organisation will put together an interview process, hiring criteria, and individual managers will start making hiring decisions. You are supposed to hire quickly, and priority number one is to hire the best people, but not necessarily the ones you need. This is deadly sin number one: not having explicit and detailed criteria as well as hands-on control over who you hire in this phase. The people who you hire now will define your company in years to come.

Many years ago, during the dot-com boom, I worked in Silicon Valley and got exposed to the cynical view that:

“A-people hire A-people, and B-people hire C-people.”

Through the best of intentions, you are now hiring (hopefully) smart people who can help you reach the goals of scaling up your company. If not, you will get a lot of people who are good, but not star performers, and then they will hire people who are not that good. Be careful, who do you need and what do you want for your organisation? Smart or best are not good hiring criteria, what will they prioritise and how will they shape your organisation?

Anyway, some of these new hires will come from larger organisations (you need that). However, this is where you risk doing deadly sin #2. As you grow, people will loose touch with management decisions, who is doing what, how to get things done, and as growth is typically chaotic, you will get dependencies across different parts of the organisation that will slow things down or even prevent people from even attempting to do the right thing. The response is always the same (and especially from people who come from larger organisations): We need more knowledge sharing through presentations, newsletters, demos, documentation, presentations, wiki pages, you name it. AND, we need processes that can help us get things done: A program management meeting, let’s hire project managers, we need more coordination meetings, who can make sure we sync up across teams, how do we make decisions, who can describe how we are supposed to do X?

You were used to having an organisation where everybody just made it happen, now, you have people who see it as theie job not to make it happen, but to coordinate, call meetings, write summaries and project plans, and structure your Jira projects. And who do they engage? Exactly the people who used to spend all their time making it happen, so now they have to spend time in meetings, report on progress, and update tickets. And they don’t think things are going in the right direction…

It is all done with the best of intentions and there IS a need to coordinate more, document things, and create some processes. And after all, this is what worked for organisation X (replace X with your favourite large tech company). The two main problems are that:

this is often overwhelming to people, and you risk them disengaging from many of the things that are not directly related to them and
you are introducing bureaucracy (i.e. a predictable process), but is it at the right place (helping you scale) or do you create barriers for getting the right things done? Also, your early generalists are bickering, this is not what they like. Where they earlier just fixed the problem, they now have to follow a process and they feel they waste their time. You risk loosing some of the star performers that enabled you to do the right things early on

So, deadly sin #2, overwhelm everybody with knowledge sharing that they don’t really know if they need and not controlling strictly where you introduce the right process, so in sum you frustrate people and slow down execution.

This is where you are about to enter the ranks of established companies, and your employees will be able to complain about how much red tape and meetings you have, and how difficult it is to just do the right thing because there are so many people who need to have an opinion. Now we are getting to the worst of the deadly sins, #3: introducing top-down processes of program management, managers, decision processes, gating criteria, and other mechanisms that are meant to get overview and control. It is in the nature of people to seek control, of course the organisation needs to be in control. The move towards the top-down control is reinforced by the people you hired. People who are very comfortable with establishing processes and deliverables to get in control.

But what is really the alternative?! Not putting in place ways to get decisions done, coordinate with others, and share knowledge will also slow down your organisation. We hear about empowerment of employees and establishing independent teams that can make decisions without consulting everybody else. But is that desirable? Can you avoid top-down control and still move forward as one company? So, should you avoid it? You want the ability to deliver on your corporate goals, you want to be able to estimate delivery dates, and you want to report on the progress, right?

De-coupling of dependencies between teams and individuals is the only scalable alternative to top-down control. It requires careful design of organisation, responsibility, and accountability.

This is really, really hard, but it is the only way you can get employee empowerment and the engineering and product velocity you want. If that is what you prioritise. The top-down control and planning may be something you prioritise, and you are willing to sacrifice velocity to get it. However, if speed is what you need, you need to organise small teams that are not dependent on top-down planning processes, where dependencies on other teams can be loosely coupled, and where the teams and the individuals are comfortable with making decisions and deliver on what they believe is the right things to deliver.

Teams will in reality never be independent, there will always be some other teams that will be impacted by decisions or that rely on what is delivered by others. Also, you want people to run in the same direction. This means that you need two things to get a de-coupled organisation:

a strong sense of strategy and purpose down to team level that guides decisions and
technical and organisational mechanisms that make it simple to understand how to interact with other teams (i.e. loosely coupled).

Designing a loosely coupled organisation is not trivial, especially when the urge to get that overview, that top-dow control, is so strong. However, if somebody pushes a change to a repository, the right documentation is updated, and the right people are notified, and they know how to change and update their own code, then you are onto something!

The key challenge is that the urge and the ways to implement top-down control are generic and well-known, while the path to de-coupled organisations is always specific to the organisation and requires careful and step-wise implementations that are tested and continuously improved. This is why the three deadly sins are so connected: the people you hire will determine whether you are able to scale through de-coupling or top-down mechanisms.

The Perfect Engineering Manager

October 2, 2019 by Greger Wedel

In my last post I introduced a practical approach to engineering leadership and the skills required in any engineering organisation.

The post must have struck a nerve as it got a lot of reads, especially on Medium in The Startup magazine where I cross-post some of my blog posts.

However, I’m sure people have many new questions on the practical side of how this works in practice. The key insight I tried to convey was that all engineering organisations are built around four distinctly different skill sets and that you need engineers who jointly cover all the skills, no matter how you distribute the responsibilities: code development skills (the developer), task priority and tradeoff handling (the backlog owner), technical architect skills managing long-term vs short-term (the architect), as well as people and team skills to keep people happy, make them work on the right things, and make sure things get delivered (the manager).

The larger the organisation, the more roles tend to be specialised. As a result, the coordination cost goes up. You typically then get elaborate frameworks and formal development processes to ensure coordination. This is a drain on time and motivation for most software engineers. On the other hand, in successful and small startups, you typically see very flat organisations where everybody seems to do everything. How then can you successfully scale an engineering organisation and keep the energy and productivity? Amazon’s 2-pizza teams is a famous approach, but reality is that it all starts with the people you recruit, so you need to start early. If you hire a project or program manager who has been managing projects and processes, but without delivering own content into these projects, you will get more projects that are managed by process, not by purpose.

And “purpose” is a keyword. You want people who are driven by purpose, not process. And here we get to the Perfect Engineering Manager. If you have one Perfect Engineering Manager on each team, focus will be on purpose and you can reduce coordination to a minimum. So, who is this Perfect Engineering Manager? In fact, he or she probably does not exist, or there are very few of them out there. Luckily, the skills can be trained. Not surprisingly given the lead-in to this post, the Perfect Engineering Manager is a person who is a brilliant coder, loves enabling and helping out others to make them successful, takes natural responsibility for delivering high-quality code into production, knows the code base inside-out and will re-factor quickly while adding new abstractions and code that will help others deliver in the future. Mentoring more junior developers comes as a given, and the natural authority of the Perfect Engineering Manager makes it a breeze to drive prioritisation, sprint planning, daily stand-ups, or any other chosen development approach. Of course, the social skill set of the Perfect Engineering Manager means that other developers love working for her or him and necessary interactions with other teams are always pleasant and smooth!

The Perfect Engineering Manager is an ideal and getting closer should be an aspiration for well-rounded, senior engineers. Not in the sense that everybody should strive to become Perfect Engineering Managers, but that one or more skills can be chosen and improved as part of developing as an engineer and a human being.

Without at least one such person, or a team of people who together cover the Perfect Engineering Manager skills, you will not have a 2-pizza team, you will have a small group of developers that rely on others to get things done. Developing great software and great products is not rocket science, but it requires systematic and careful grooming, development, and attention. In too many organisations, software engineers are left on their own without much help or guidance on how to grow their career, job satisfaction, contributions, and as human beings. The Perfect Engineering Manager is of course instrumental in facilitating and making the right decisions to make sure that the engineers on his or her team are not in that unhappy group.

A Practical Approach to Engineering Leadership

August 17, 2019 by Greger Wedel

In my previous post on the new engineering leadership skills needed in a devops world, I called for a more systematic approach to developing leadership capabilities in engineering. But I gave no answers. In this post, I’ll share my practical approach to developing leadership capabilities in an engineering organisation.

When developing great software, there are some fundamental questions all engineers have:

What can I decide?
Who can make decisions on things I cannot decide?
Who do I need to talk to or involve when resolving an issue?
Who determines the targets for what we make, how we make it, and how we go about doing it?
What is expected of me and what kind of rewards do I get when I meet expectations?

In a small startup team, you don’t really need a lot of these things explicit. The goals, actions, and rewards are mostly shared across the team, and everybody contributes where they see a need. The entire team has the same priorities, the tradeoffs are decided upon continuously, and changing course is done quickly.

However, you cannot grow much as an organisation without introducing slightly different priorities in different teams or roles. You are starting to introduce the cost of coordination. For a while, you can align everybody around the same priorities through some kind of coordination process or meeting that gives you the “top 5 things to work on this week”. But to allow rapid growth you need to at some point allow different people or teams to focus on different priorities. I have seen numerous approaches to solving the resulting coordination problem, and most either introduce too much process or too little.

There is no single answer that works for everybody in every situation. However, if we go to the fundamentals of how to build great software, there are four different areas that always have to be covered in some way:

Build high quality code, designs, and other tangible stuff
Knowing what to make to solve the customer’s problem
Knowing how to technically solve the problems and deliver something that will work over time
Knowing how to go about making the tangible stuff like code and actually delivered what’s agreed upon

The skills that go into each of these four areas are very different. And the more people involved, the more coordination and collaboration is needed to make sure everybody is pulling in the same direction. Ideally, you want everybody to be 100% productive, but that is not possible as long as there is some kind of dependency between what people are supposed to do. How do you get the productivity percentage as high as possible?

Typically, each of these responsibilities translates into one of four roles (not individuals):

Developers build stuff
Backlog owners manage the priorities and the definitions of what is to be made
Technical architects determine how to technically solve problems
Engineering managers lead people, the process of development, assign the right person to the right task, and ensure velocity

These are roles, not individuals, so many engineers will wear multiple hats. Also, there is one missing part here: the service or product. Each organisation has to break down its services and products (and thus code bases) and group them in a sensible way. Once that is done, it should be possible to identify the developers, backlog owner, the manager, and the architect for each. An engineer can be a developer on one code base/service, an architect on another, as well as manage a deliverable.

This break down into modules and components is absolutely critical to the productivity of your engineering organisation and subject to Conway’s law.

Unless you match the identified ownerships and roles with your modules and components and get that right relative to your organisational structure, you are introducing inefficiencies in your software architecture and how people communicate. This is a continuous process and needs to be adapted to the changing needs with respect to both features and the underlying system architecture.

Going back to the questions at the top of this post (that most people have in a software development organisation), it should now be possible to answer the questions. You don’t have to go through a central manager or (..!) a committee or a process, you can reach out directly to the person in charge who can make a decision. E.g. if you are a backlog owner and you need more engineers to work on your backlog, you can talk to a manager. If you need an evaluation for how to implement a certain functionality, you talk to the architect. If several components need to communicate through an API, you know exactly whom to invite to discuss this.

You thus get an organisation that can resolve most day to day matters without central coordination.

Typically, you will have triplets of lead people for each of your main components: a manager, an architect, and a backlog owner. The development team can thus be productive by getting help from each of these three. Ideally, all three roles/people are co-located physically for highest productivity since high velocity requires quick and frequent touch points.

There are two other roles missing in this picture: if the team is a product team, you need a product manager role (who could also be the backlog owner, dependent on skill set). If your component or product has a UI, you should also have a UX lead who can work with the backlog owner and product manager to get clarity on the deliverables. See also an earlier post on the backlog and the product owner role.

This model is also good for training and career development. There are four areas where skills can be built dependent on individual interest. As engineers grow from juniors to become more senior, they will take on more roles and expand what they take ownership of. Some may want to take on manager roles and lead teams. Some may want to deep dive in a very narrow technical area and become lead architect.

Remember that any changes in an organisation will take time. You need to repeatedly clarify the expectations, the opportunities, show that real authority to make decisions come with these roles, and systematically encourage people to take ownership. And if people are uncertain, you need to exemplify what concrete tasks and actions are necessary and expected.

The reward is an empowered engineering organisation that can deliver great software with high velocity!

The Program Manager in the Agile/DevOps Organisation

March 24, 2016 by Greger Wedel

This is another post in my series on transitioning from waterfall to agile and devops processes. This post covers the program manager role, a rather unassuming role in some organisations, an important role in others, but nevertheless a role where the right person can potentially make a big impact. There are a few different types of program managers, so let me be clear on the type I will discuss here. Microsoft uses the title for product managers with a horizontal responsibility, i.e. functionality that should go into multiple products. Other organisations use the title for a role that is similar to a product manager tole. Typically, a program manager of this type has an independent responsibility for business results within a certain area. A very different program manager role is a project manager for a deliverable that crosses multiple independent products or teams. Often, a program manager of this type will work with multiple project managers, one for each product involved in the deliverable, but does not have an independent ownership of business results and thus becomes more like an advanced project manager. In Cisco, program managers are often of this last type (and clearly lacks the first type), and alas, some larger products may have their own program manager working alongside the project manager to manage and coordinate outside the product team.

I may cover the consequences for organisations not having the “program manager with business ownership” role in a later post, but here I will cover the “program manager as an advanced project manager” role.

Of the many roles in a development organisation, the program manager role may be the role that involves the most painful change in mindset and daily tasks when a large organisation moves to agile and devops. The traditional tools of a program manager is no longer of much use: defining deliverables, setting and tracking milestones, managing dependencies, measuring progress against the plan, reporting status and escalating issues to stakeholders… But why is that? The reason is that the traditional program management model and associated tools assume a planning horizon (e.g. ship within 6 months), a clear view of what is “inside” and “outside” of the boundaries of what is managed (e.g. we need to ship a and b and rely on c and d to deliver x and y for us to be successful), a desire and need for a pre-deterministic plan for what is “inside” (e.g. we are going to ship a,b,c, and d), and that any deviation from the plan, including “outside” changes, are bad and should trigger a corrective action or some kind of mitigation.

ALL of this is wrong for agile and devops! For one, changing plans is desired, that is called learning. Also, the boundaries of what is “inside” and “outside” may be shifting as the teams learns and gets feedback, and reacting to change is something that each member of the team should do immediately. Planning should be done iteratively based on learnings and changing requirements, and what is actually shipped is not fixed as teams commit to a prioritised set of tasks (backlog), not a fixed set of deliverables.

Alright, so the program manager role is dead, is that what you say?! Yes, the OLD program manager role is dead, really dead. However, the reasons for why the program manager was needed are still there: ensure that multiple teams and projects with independent execution paths coordinate priorities and execute efficiently so that real cross-product value can be delivered to the customers with the right quality!

Can program managers re-invent themselves? I really don’t know, some of them may be stuck in old ways of doing things. There are of course many flavours of agile that companies may choose to adopt, like SAFe, where you get clear definitions (http://www.scaledagileframework.com/rte/) of what a program manager is supposed to do. The problem I have with such frameworks and role definitions is that activities, tasks, and goals are often defined in terms of artefacts and ownership of activities defined by the framework. Thus, the role quickly becomes statically defined, which is contrary to the whole point of agile: constant improvements reacting to learnings. By the way, as a rant on the side: that is probably also the biggest problem with frameworks like SAFe in general. They capture some important learnings and concepts, but become stale and not a framework for change.

I really like to get to the core of what we are doing and what we are trying to achieve. E.g. “Manage risks and dependencies” is an abstract statement, and it involves too many implicit definitions of what a risk is, how you manage it etc. But looking at the problems I have seen in agile and devops, a typical program manager personality can contribute buckets to some of them (but constantly change HOW to contribute to them as the organisation evolves):

1. What do we want to deliver and when? In large programs, multiple product managers need to agree to a set of ordered business priorities. Without clear and shared priorities across the products that are to be delivered together, we get absolutely nowhere. This is the role of the product managers (or rather ironically the role of the other type of program manager in organisations that have those). As covered in the post on the backlog, I believe in a true and single owner of a backlog (delivering on the priorities), and the backlog being defined as the task list a development team uses in their iterations. When a team of product managers come together to define shared business priorities, the team of product owners need to come together to tie their backlogs together (as described in my backlog post). A group of peers with priorities outside the group has always problems with closing decisions because conflicting product-specific priorities will interfere. A program-level product manager (aka solution product manager aka that other type of program manager.. puh!) can be used to drive the program/solution priorities and resolve the conflicts between the program-level and product-level priorities. Likewise, a program-level product owner can work with the product-level product owners to tie the backlog items together so that co-development can be done (where co-development means actual coding done in the same iteration with developers talking to each other). This program product owner role can be filled by a program manager (assuming some technical knowledge).

Solution Product Manager ——— Solution Product Owner ——— Solution Program Manager

|| || ||

Product Managers Product Owners Project Managers

The flow is thus: define and prioritise shared business priorities —> define interlinked product-level backlog items to deliver on business priorities —> plan co-development where needed. I tend to talk about “backlog items that are flagged for co-development”, which basically means that there will be another (named) developer in another team (or more) who should be involved when writing and testing the code for that backlog item.

<Whining rant> When writing this, I really feel the urge to go back to a startup environment and the simplicity of making decisions and pushing things out the door… The less people who don’t code, the higher feature velocity you get. But then again, you typically have just one or a few products and limited coordinations needs… </Whining rant>

Another important thing in devops is metrics and A/B testing. This input should be continuously brought into the business priority process. Often there is a breakdown in communication between engineering (doing metrics and A/B testing) and product management (setting the priorities). I have yet to see program managers really making a difference here, strong architects and engineering leads tend to basically take ownership of parts of product/service improvement, but I believe this is an area of real value creation where a good program manager can make a difference.

Another important aspect of program-level work is to set the right targets and make sure engineers can focus on getting things done. It is important to go to the right level of detail on the program-level of what a specific program backlog item really means. Getting to the right level of detail on wireframes, prototypes, and requirements through demos, transfer of knowledge sessions, and white-boarding is important to make sure that the engineers from all of the teams involved in co-development have a shared understanding of what needs to be done. If there is one person responsible for driving this on behalf of the product owners, things seem to move faster.

Of course, another important part of this phase is to get the architecture right, engaging the senior engineers in each of the teams and preferably with a senior architect with program-level knowledge and ownership to lead the process. In Cisco, there is a separate technical leadership career path where director-level technical leaders (principal engineers) are expected to define and supervise architecture in larger programs and also across programs and business units (I’m btw one of those).

2. How do we tie together iterations to do co-development? Once the priorities have been agreed to on the program level, the engineering managers and product owners need to coordinate and identify when and how to do any needed co-development. This will include backlog grooming, developers planning an iteration together, sharing builds, and coding together possibly on a day to day basis. Two and two teams can manage this nicely on their own, but if three or more product owners and engineering groups need to coordinate, there is often the need for a facilitator. Again, this is more of a solution-level product owner type of role, not a program manager, but a technical program manager would fit nicely in this role.

3. How do we bring together the products, validate that iterations delivered planned outcomes, and qualify the process towards release/deployment? Once an iteration has been done, there is a need to bring builds of each of the products (or services) together to do testing and validation of the deliverable on a solution-level. In a true devops environment with continuous deployment, you will get this validation quickly, but if you deliver products and not devops services, you will need some solution-level live test environment. Solution-level testing needs to be done (again, preferably done as part of the iteration…), and if everything is ok, the builds can be deployed to a wider audience, either internally or to early trial customers. This is a critical stage as solution-level bugs will show up. There is a need for a strong person managing the maturity of features delivered and determining whether to allow the features to go beyond the initial solution test bed. In a devops approach, you will have developed features with #debug flags and feature-toggles, and features can be turned off if they are deemed not mature enough for deployment to a wider audience. The program manager can be an excellent watchdog in making sure that the right quality is achieved before releasing features to a wider audience.

4. Once a development cycle reaches an end and we want to ship, how to get things out the door? We want engineering to have a rhythm, a two-week pace that will give us some small, new improvements and features for every iteration. Ideally, engineering will not miss a beat when the products are released. However, more likely there will be a need for one or two hardening iterations at the end of a development cycle to clean up bugs, improve stability, and fix remaining issues. Once this has been done, all the code repositories should get a release branch, and the bulk of the team should pick up new feature development for the next release, while some time should be allocated to fixing bugs on the release branches and merging back to trunk. Dependent on the products/services, this final step before releasing products to customers can be long or short, but preferably it should be as short as possible. Typically, tasks will be final stress testing, solution-level manual testing, documentation finalisation, knowledge transfers to customer service people and sales engineers, release notes, bug scrubbing, as well as other steps to ship the software. This whole process play more to the traditional strengths of a program manager who is an obvious candidate to own this step of the process.

As you can see, with references to artefacts like backlog and list of business outcomes and to various roles, I get dangerously close to defining a framework of my own 🙂 The difference is that I here describe a(n ideal?!) system of people and deliverables that MAY work, but that must be adapted to people’s experiences, preferences, and beliefs through an agile process where people bounce off each other and change together. If not, the deficiencies inherently found in any human activity will settle and some important things will continue to be dropped, people will have overlapping responsibilities, and teams will work in slightly conflicting directions without the continuous improvement that agile promises.

But in sum, this post is an argument against the opinion that agile and devops mean that you don’t have to do any planning, documentation, or process management. On the contrary, some of your planning and process management need to be even more diligent than in traditional waterfall. So, don’t sack your program managers before you see how they can adapt…!

The Changing Development Organisation: The Software Architect

July 20, 2015 by Greger Wedel

(intro to this post on devops)

It has been a busy few months. Since my last post I have transitioned to a new role as part of day to day development (see linkedin.com/in/greger for more details) in the Collaboration Infrastructure Technology Group (UC, telephony, telepresence, conferencing, etc) in Cisco. Earlier, as part of Office of the CTO, I influenced the medium to long-term technology and architectural strategy, but as I had limited involvement in the day to day operations, it was a challenge to develop architecture that bridged near-term goals with longer-term strategy. Now, I’m less involved in long-term technology strategy, but on a daily and weekly basis involved in designs, decisions, and priorities made as part of development. Quite refreshing and comes with a bunch of interesting challenges!

If you have followed my blog, you have probably understood that I see architecture as something tightly embedded in day to day development, but more than just the sum of tactical, short-term steps. Also, you have probably detected my desire to see product management team and architects partner and work on the same thing: identify the whats and hows of building great products and services. Agile has for a long time changed my organisation and the last year or so, devops processes have really started to impact how we work. We have hardware projects, traditional waterfall product releases, and cloud services. This means that in my technology group we are currently using various flavours of waterfall, agile, and devops processes in parallel. It puts a strain on coordination and especially challenges existing roles in the teams. This post is about some of these changes.

Classic product organisations used more or less waterfall processes, and many so-called agile development organisations are using agile terminology to execute on a mostly waterfall mindset. However, with an increased focus on devops processes, the combined product/service organisation is facing another set of challenges in combining everything from hardware products and their software loads through legacy, monolithic, hosted applications to devops microservices. Tying these together to deliver robust, scalable services that offer great user experiences is a challenge along many dimensions: culture, people, competence, programming languages, tools, project management, goal setting, measurements, management, and roles.

One particular challenge is that not only does what we do change, our roles have to change as well. This is easy in a small organisation, but in a big organisation, the expectations other people have to what you are supposed to do is very important to your ability to get things done. So, let’s say I’m used to architects writing specs, then I will likely continue to ask for specs. Other people may not think an architect should write specs anymore, and they will ask for other things and maybe ignore the specs. This creates frustration and tension.

I believe most people in the industry are starting to understand the fundamentals of how devops are changing the roles of the development team and the operations team, and that they basically need to become one integrated team (with some new roles and new definitions of existing roles). However, the role of the product manager is also changing in the process, as is the project manager’s role, the engineering director’s, as well as marketing, sales, support, and partners. They no longer have releases to plan around and need to change how they work. I may cover some of these other roles in later posts, but in this post I will concentrate on the architect’s role.

First of all, a few words on what an architect is. In my understanding, an architect is a developer, a member of the engineering organisation, with a special responsibility for understanding the short-term and long-term business and technical requirements and making sure that the right technical deliverable is delivered at the right time with the right quality. Sometimes this is to get it done and deliver something less ideal from a code point of view. Sometimes this is to tell the product owner that some labour-intensive refactoring needs to be done in order to deliver a feature. My personal view is that a good architect also truly understands the value of the product to the end users and customers, and that he or she is a partner to the product manager or owner in defining and evolving the product. An architect can be a senior member of a scrum team (in Cisco, these are called technical leaders), or a more dedicated role that covers multiple scrum teams (senior technical leaders or principal engineers). In larger organisations, system architects typically span multiple products or services to create functioning solutions. In my view, regardless of scope, to be effective, the architects should be embedded in the engineering organisation. Some architects lead development teams, others have a more standalone role. However, the most effective teams seem to always three roles covered when decisions are made: the product owner (representing the vision and the problems to be solved by the product), the architect (representing how to deliver with the technical integrity of the product in mind), and the development manager (representing the developers, and making the trade-off decisions).

The architect is thus a member of the product leadership team. And with agile and devops, this role becomes even more challenging with the need to engage in more areas. Here are the some of the main areas of challenges:

Software architecture and code structure: There are two big challenges I want to mention here. First of all, devops requires a modularisation into micro-services, and getting the right grouping of functionality and defining clean, scalable APIs are critical. Second, the whole thinking of how you develop a feature or a set of features need to change. Every time you commit code, you cannot break anything. This influences how you architect the feature, how you test it, and you need to think in terms of A/B testing, feature toggles, and metrics, metrics, metrics(!), and not in terms of feature branches and unit and integration testing.
Software development processes: An architect needs to engage in code repository management, branching strategy, build ande delivery pipeline, test automation, A/B testing, feature toggles, 360 reviews, and much, much more. The architect needs to police the process and make sure that scrum teams working on various micro-services have the same view on the process and don’t break the builds. Very few other people will have the code knowledge and competence required to identify where the issues are and address them. The architect becomes critical to making sure that the devops machinery keeps working.
Requirements and product management: A shift from waterfall to devops is a shift in power from product management to the engineering organisation. Where product management used to ask for features based on market knowledge and research, a high speed of introducing features, doing A/B testing, and measuring the impact and quality relegates all product managers (except those who are very technical) to focusing on the go to market aspects of the service, not the requirements. The architect’s role in this becomes critical as the architects are instrumental in defining what to measure, and what you measure, you improve (whether this power shift has to happen, is a separate topic…)
Project management: Traditional project (or program) management should totally disappear. The software development process, the build pipeline, the metrics, the scrum standups, the scrum of scrum meetings, kanban boards, and the definition and usage of the APIs across the services, together with the automated tests together define and drive the project. The traditional tools of a project manager, like Gantt charts, dependency tracking, weekly project status meetings, read-outs to executives, escalations, and decision meetings based on analysis, are of absolutely no use in a properly executed devops process (note that badly executed devops processes will give ample room for such old tools and processes…). The architect needs to engage in the overall execution of the devops project to ensure that there is a constant evolvement of proper architectural frameworks, guidelines, automation, gates, micro-service designs, API/protocol designs, and much more.
Architecture: Devops does not mean architecture work is not necessary. On the contrary, as the architecture is constantly and organically evolving, the architecture that emerges can either be a disaster or a robust construction. However, the “ivory tower” architect is history as the emerging architecture emerges both with and without a plan, and it can only be shaped by having constant touch with what is being developed. A spec or design document will be obsolete within a week.
Management: I see a lot of very competent, senior architects who are brilliant at designing a flow or a new protocol, but who are lacking in managerial skills. Running efficient meetings with action items, pro et contra evaluations, closing decisions, setting deadlines and targets, chasing deliverables, and driving consensus are just examples of typical managerial skills that becomes more important for an architect.

There is so much more to how devops is changing an architect’s work, as always, the devil is in the details, but these categories have been helpful for me in understanding some of the challenges in evolving architects towards “devops ready.” The shift is similar already with agile development processes, but becomes more visible with devops.

Do you believe there are other important things that impact the architect’s role in devops? I’m happy to hear from you!

My next post covers the importance of one shared backlog backlog with all the actual work items visible, as well as the product owner’s role. Not directly focused on devops only, but relevant to devops as a good backlog grooming process is essential to getting anywhere.