How to Scale a Product Engineering Organisation?

There is plenty of good product management literature out there, but if you want to scale a large product engineering organisation, you mostly find the agile scaling frameworks. One of the most visible advocates for building innovative high-tech products that delight customers, Marty Cagan, slashed out at these frameworks in June 2021 offering little hope for their usefulness outside the “not-so-innovative” corporate world. So, how do you then approach scaling great product development?

In Marty Cagan’s EMPOWERED, product leaders take leadership responsibility for the PRODUCT, not just the people who report to them. This is independent of whether you are a product manager, tech lead, engineering manager, designer, or CTO. Terese Torres’ Continuous Discovery Habits shares advice on how to integrate product discovery techniques into the daily work of the team. Great stuff, but if you are like me, as an engineering leader or CTO, you miss guidance of how teams actually should work together to build solutions to problems that are bigger than what a single team can tackle. For many problems and as your organisation grows bigger, you need to innovate and experiment across multiple teams in a (somewhat) coordinated fashion. You need to build an organisation and a culture that support the product trio in the teams (product manager, designer, tech lead), but that also allow the teams to build together.

You see dependencies everywhere, deliverables are slowing down, and you are asking yourself: Do I have the wrong team composition? Do I need to reorganise? How can I avoid the disruption that follows? Have we failed in building the right architecture and decoupling between teams? Do we have the right architecture? Or don’t we have the right roles, processes, and culture in our organisation? Why are we not able to scale how some of the teams are able to work to the rest of the organisation? Are Cagan’s methods really suitable beyond startups? Should we start doing some type of scaled agile?

The missing piece in Marty Cagan’s product development story is HOW the products are built. Quite natural as this responsibility is with engineering, not product management, and Marty is a product manager. You need to systematically invest in evolving your technology architecture and key enabling technologies needed for longer-term velocity. If not, you are essentially trading short-term velocity for long-term velocity, and you get a tangled mess that will slow down the entire organisation. This is true for all engineering organisations, not only those who follow Cagan’s approach to product development.

Three Main Anti-Patterns

I have seen three main anti-patterns emerge out of a successful start-up product organisation that threatens velocity and potentially the entire company’s existence. You believe you have product-market fit and you are scaling up sales, but your engineering organisation slows down to a grinding halt. You typically have one of these situations:

1. You have a monolith, and your engineers are increasingly stepping on each other’s changes and spending time trying to figure out the impact of a code change;
2. You are out of the monolith, but you have no coherent architecture, and all the teams are solving different parts of the same problem in silos, thus quickly making MVPs for their part of the product. But, without the MVPs connected to the same underlying enabling functionality, they quickly slow down when trying to bring the MVP into a unified product offering;
3. You have too much refactoring and enabling-functionality work going on , and engineering is busy re-building the successful early-stage functionality into “something that can scale”;

In all these situations, you find yourself in what I refer to as “the valley of no apparent progress“. Sales is becoming increasingly frustrated (especially in a B2B company), product management is pushed to deliver a committed roadmap, and the stress-levels go up in the engineering organisation. How to get out of the quagmire and again start churning out delightful solutions to your customers’ problems?

The Valley of No Apparent Progress

Ironically, the first time I found myself in a big “valley of no apparent progress”, was not in a start-up or scale-up, but in Cisco. Back in 2010, Cisco acquired TANDBERG, and I found myself in midst of the biggest strategic topic of the product and technology integration process: how to make it possible for Cisco and TANDBERG meeting room video systems to participate in the same meeting?

After the TANDBERG acquisition, Cisco had two products mixing together audio- and video-streams into a video conference: the Cisco TelePresence Meeting Server, and the TANDBERG TelePresence Server (I blogged an entire series about the technology, starting in 2014). Their architectures and implementations were totally incompatible (so anti-pattern #2 above). The obvious customer need was “merge the two products into one so everyone can participate in the same meeting”. The old TANDBERG and Cisco product teams were eight timezones away from each other and there were two independent organisations all the way up to the Vice President level. The product leadership wanted the absolutely worst possible thing: make the two products talk to each other, but keep them separate (“don’t kill a good revenue stream”). Here, Cisco had the opportunity to leapfrog the industry and start building in 2010 what you can see today in Cisco Webex and Zoom video conferencing. It chose not to.

However, this is how I started to engage actively with Conway’s Law, and what was later coined the “reverse Conway’s manoeuvre”: You establish an organisation to reflect how you want the technology architecture and products to be built. I spent many years navigating the Cisco organisation to try to influence the collaboration architecture and what the teams were able to build by establishing the right teams and boundaries between them.

Conway’s Law Is Real and Ignore It At Your Own Peril

Cisco’s underlying problem after the 2010 acquisition was actually the same problem that most scaling organisations face. As you grow, there is a natural tendency to split the product development organisation into three: professional engineering management, senior technical leadership/Office of the CTO, and product management. Cisco had TWO such structures right after the acquisition. However, the reality is that there is a tight relationship between the product(s) you are able to build, the technical architecture that gets built, and the teams’ compositions and ownerships. What I found at Cisco was that most engineering managers (and definitely the Vice Presidents) were not technical enough to understand how the organisation they chose impacted the products that were actually made. With organisational decisions being made for other reasons than building great products, the products under one Director could be great, but anything that needed coordination across multiple product groups was very complicated to make.

This misunderstood separation of duties between architects, engineering managers, and product managers where managers have the final say on the organisational structure becomes even more problematic with modern SaaS architectures and micro-services that heavily rely on a coherent architecture.

Let’s rewind back to frameworks for scaling agile: reality is that they are made for organisations like Cisco. They assume separation of duties and establishment of processes that reinforce the separation of engineering management, architecture, and product management. This is why the book The Unicorn Project from 2019 resonated so well with many product people: it shows how bad things can get with unchecked evolution of organisational structures. Great products are built as a continuous stream of small decisions that make the organisation, the teams, the architecture, the technology, and the product evolve together. Product, engineering, and architects are not three separate organisations, they are ONE, but left unchecked, each function will evolve and optimise in isolation.

Gene Kim, the author of The Unicorn Project, had just one year earlier been the co-author of a more important book, Accelerate, that thoroughly documented how effective DevOps processes can be. In Accelerate, they showed how to measure velocity in various ways, not whether you build the right things. But it’s exactly this establishment of a continuous flow through the organisation that DOES NOT go up and down the management chain that is key to making products that excel.

Getting to Continuous Flow

Be warned: It’s a lot easier to just adopt one of the scaled agile frameworks and continue to be the manager you have always been. Making a product development organisation that is able to sustain continuous flow requires a totally different set of leadership behaviours.

So, just a quick status quo: your organisation has been successful and you have built great software that has solved real customer problems, they have paid for your product, and you have hired more people and are ramping up to solve even more problems for the same customers and scale the product to be able to support 10x, 100x, or maybe 1.000x your current customer base. Maybe you were just hired into an organisation like that to “strengthen the team”? The way you have built product so far is breaking down. You used to have 30ish people working together as a group, ignoring roles, just getting things done, now you are beyond 50, maybe more, and the dynamics are changing.

Knee-jerk reaction: split out a professional product manager group, hire engineering managers, and split ownership of the code into teams. But then what?! How does this organisation make decisions? How do you iterate and continuously improve the understanding of the customer and the customer’s problems? Who should interact with whom and when? How do ensure that you get a “highway of communication flows” between customer and developers (and everyone in between) instead if a trickle through product management?

The more leaders you bring in, the more perspectives you will have on what the “right way of working” is. Your ability to grow great culture, product discovery, and product delivery processes will exponentially DECREASE with each additional leader you add UNLESS you have the foundation in place and recruit leaders who share that foundation. There is not one right way, but there are many wrong ways, and you need one consistent approach, not a mix-up of different product development philosophies. The larger your organisation gets without this foundation in place, the bigger the change management effort.

1. Identify (potential) product boundaries and establish cross-product interfaces

So, how do you counter the knee-jerk reaction and the addition of new leaders? Start by realising that you don’t make organisations that collaborate, people collaborate. People need trust, time to build relationships, competence, and learn how to be effective together. And they need as few people as possible around them to establish this. Adding hierarchy and formalising roles will only lead you to scaled agile (or worse) unless you systematically build a culture and an organisation that can efficiently build product both within a team and across teams. You thus need to build organisation from the bottom up, not top-down: Focus on individuals and teams.

You will make the change management process a lot simpler by splitting your product development organisation into smaller, independent groups that can interact with other teams the way they would interact with somebody outside your organisation: through an API, good documentation, and clear promises made on how the services will behave (think: cloud services). Amazon (AWS) has taken this to the next level: all their teams interact with other teams as customers and providers through decoupled interfaces. They have a few cornerstone services that most teams use (IAM, S3, EC2, etc) and then use tooling to ensure that independent services appear uniform outside AWS in APIs, SDKs, UI, documentation, and other ways.

AWS’ model may not fit your needs, it comes at a cost, but it scales really well. But by identifying one or two products or modules that only a sub-group of customers uses and split those out, you reduce the number of people who have to interact and agree to make progress. Typically you will find that there is value-added functionality that can be carved out as separate, independent groups.

Then, establish the interaction interfaces BETWEEN these product groups. Are some libraries suited for open-source style collaboration? Do you have SaaS services that can offer APIs and documentation? Should product managers treat the value-added groups as their customers and collaborate closely to understand their problems and needs? If you need weekly meetings across product groups, you have probably not cleaned up the interfaces properly.

Also, are there common tasks that could benefit from an internal developer platform? A self-service platform where tasks are automated, templates are pre-created, and resources can be provisioned easily by developers? If that is the case, all investments in the internal developer platform will accelerate everybody. The customers of the internal platform are all the development groups, so make sure you have a very technical product manager to prioritise in the internal platform. These investments should complement a focus in the teams to prioritise improving developer velocity on a continuous basis.

Similarly, you may have things that need to be consistent across the product development organisation, like change management, how you make your software and services secure, how to instrument services, logging, tracing, and how to scale services globally. If you do, you may evaluate a separate security organisation, a reliability team, and dependent on your needs, individuals or teams who are responsible for setting standards, supporting teams, and making sure that you get the desired consistency. Make sure that these supporting teams focus on empowering the development teams to deliver consistently and according to the set standards. They should not become teams with independent agendas and deliverables.

Do you see how integrated these evaluations are with the architecture, technology strategy, and the development processes? In order to have the right discussions, your engineering managers need to be deeply technical and understand the specifics. You also need to empower your senior engineers to take lead in transforming the technical architecture to support the loosely coupled interfaces that you are establishing through these organisational changes. Bring with you the product managers as well. They need to inform the decoupling as you are now making decisions on which things will be hard to do (i.e. coordinated deliverables across product groups) and where you will get high velocity (within the product groups).

2. Establish a Product Axis Defining Product Group Engagement

Notice how Conway’s Law was used above to define a smaller product organisation that will result in a cleaned up architecture and make sure that each group can operate on its own, independent of other groups. Matthew Skelton and Manual Pais have done great work structuring how to use Conway’s Law and organisational structure to establish sound architectures that support efficient teams. Their book, Team Topologies and their website is worth a deeper look if you want to go deeper in a structured way to analyse your teams and organisation.

Both Team Topologies and SAFe, one of the scaled agile frameworks, use the concept of “value streams” and emphasise the importance of organising around value streams. What is important to understand is that this is all about people and the dynamics between people. By identifying and organising around an activity that creates the most value for customers, you reduce the number of people, stakeholders, and teams that need to be involved in each value stream. This is essentially what we did in step #1 above (organise around product boundaries).

For all the product groups, you should strive for consistency in how they engage with sales, support, marketing, executive stakeholders, and not the least customers. I refer to this as the “product axis“: the optimised value stream from users and customers all the way through your organisation and to individuals writing code. Typically you need established feedback loops (not one-way!) with sales, marketing, strategy department (if you have one), executive leadership, customer support, sales engineering, customer success, professional services, and sometimes more. For each feedback loop, you need broad engagement from the product development organisation and named individuals in each part of the organisation who talk to each other daily or at least weekly. The product manager is responsible for managing the product axis and making priorities and decisions, but the fidelity of the information and insights that flow across must be as high as possible. The less fidelity you have in the feedback loops, the less awesome product you will get.

Below is how I have represented the product axis at Cognite (where I currently work).

All these feedback loops generate signals and learnings into the product development organisation. The engineering managers and product managers should be responsible for ensuring that the right people are capturing the learnings and feeding them into the product strategy, discovery, and delivery processes. Do not confuse the product axis with a tool where customer requests, asks, or stories are captured. Such a tool is great to organise and structure where to pay attention, but you cannot build a great product based on descriptions in a tool.

3. Search for Energy and People Who Believe In Something

In each of your product groups, you now need to identify how to set up your teams. Again, value stream is an important concept: where do you generate most value for your customers? There will always be trade-offs, so you should optimise for where the most value is built. It will also be dependent on the evolution of your product(s). Most often, you can find people, a product manager, a manager, a senior tech lead, or a senior engineer/architect who shows passion and a vision for how to deliver exceptional value to your customers. Listen carefully, and build value streams around these people. It can be focused on a specific application or user journey, a foundational technology capability, or a substantial barrier to value creation like internal tooling. Free them temporarily from the current organisations’ expectations of roles, deliverables, and processes, but give them a few simple ways to measure success that the team can share. Focus on outcomes and things that can be measured. Let them define these themselves and give them a stamp of approval.

You are betting on people, give them resources and time, but not totally free reigns. Expect them to come back and report on progress against their success criteria and be prepared to help clear their path as they are butting against perceptions of “how things should be done”. You are energising the organisation, not only this team, so also make sure that you also identify value streams that are “main stream” and better understood. Determine if they have the right-sized ownership, not too much, nor too little, that the right people are assigned to the team, and make sure that their priorities also drive other teams’ priorities.

How do you identify value streams? As described above, a value stream can be aligned around a product or a value-added module. But it can also be aligned around a user journey where you need focus and knowledge on how a single type of user persona traverses your various services, products, and functionality. Here in these “journey teams“, you want to staff up on designers, user research, and most often front-end developers, but also back-end developers necessary to iterate fast in discovery of problems to solve and deliver great solutions. Another value stream can be “application teams” focusing on a specific application delivering narrow value to a specific user group in a specific problem area. If an application team relies on backend teams that also service other application or journey teams, you may want to allow the application teams to develop their own backend services that unblocks their velocity, but that also should be used as prototypes for inclusion in other backend services if they turn out to of generic value.

Finally, in more platform-type SaaS services that are built to enable a broad range of applications and functionality, you may have “capability teams” owning backend API services that in themselves are primary value streams. For AWS, EC2 is such an example. Capability teams typically focus more narrowly on a technology enabler and see themselves as creating building blocks for other teams. The product manager and designer of a capability team focus on how their services simplify the building of more complete end-user experiences by other teams, either internal or external.

SIDENOTE! In Team Topologies, journey and application teams are categorised as "stream-aligned" teams. Other team types they use are "platform teams" (internal developer platform) and "enabling teams" similar to "supporting teams" mentioned above. Also, they define a so-called "complicated sub-system" team type. These are teams with highly specialised competencies needed to build intellectual property in a special domain. In my experience, beyond initial bootstrapping, such teams should be organised either as internal platform teams (see step #1) or as key maintainers of an internal open source library (accepting PRs from others). If not, they tend to become major bottlenecks.  

4. Structure value streams and align priorities

It is important that the main value streams drive the priorities of other teams where there are dependencies. You need to make clear what the overarching priorities are, so that the teams that typically sit in the “backseat” don’t try to drive priorities on their own. A backend API servicing a single application, e.g. Superhuman (with clients for iOS, Mac, and Windows), is not its own value stream and should thus sit in the “backseat”. However, an API service offering speech-to-text in multiple languages is a separate, clear value stream that may have customers both internal customers (like Google) or external customers.

Typically, journey and application teams should build functionality based on a collaboration between a product manager, a designer, and a tech lead. This is leaner, delivers better solutions to problems, and spends less resources than teams that see themselves as servants that deliver features based on a feature requests from other teams. This methodology is what Marty Cagan evangelises. The closer you can organise your teams to this model, the more ownership and the better results you will get. This also holds for internal platform teams who then need to see the internal development teams as their customers.

If you need multiple teams to align their priorities because you need to deliver on an important priority across the product group, you have two options: either you allow the customer-provider model to work between the teams (accepting that external customers may have higher priority sometimes) or you establish a few high priorities or themes across the product group with clear goals that should trump individual teams’ priorities. You should be careful with this latter approach as there is a tendency to elevate too many things to this level (more than 2-3 such priorities concurrently and you suck up all capacity). You need to make sure that you allow the empowered teams to continuously discover and deliver between themselves, not driven by top-level priorities that cascade down. You can quickly end up with disempowered feature teams.

How to make the teams collaborate well? If you have time to use the fundamental dynamics in a product team doing discovery with product manager, designer, and tech lead, you should always use that approach as you long-term will get the best product this way. However, in early discovery, hard commitments to customers, or in special cases where a priority is super-critical to get right across teams, it can make sense to fast-track deliverables (at the expense of less validation) by using the mechanism of “interlock”. Interlock can be partial or full, with the latter meaning that two or more teams align their discovery, planning meetings, stand-ups, delivery, and demos as if they are one team. Partial interlock can typically be implemented by having one team member participate across teams and allocating one teams X % capacity to deliver on a hard commitment to the other team. Partial interlock can be facilitated through definition of APIs/interfaces between the teams and the use of mocking, e.g. using an approach like pact.io.

A word of warning: be aware of the number of people in your organisation whose job it is to plan and follow up, not build product. These people have titles like project manager, product owner (managing features received from someone else), program manager (unless these are product managers like in Microsoft), coordinator, and so on. Ask the question: is this person focused on what is built or the process of how it’s built? People primarily focused on the process of how it’s built will almost always pull your organisation in the wrong direction and pull all the most important people into planning meetings they will hate.

How to Know That You Are Making Progress?

In a non-trivial organisation you will have different types of teams, and some teams will be able to execute independently as a value stream, while other teams will have strong and painful dependencies on each other. Especially is this true when you are starting to use the reverse Conway’s manoeuvre as there will be dependencies between teams until they have architected better decoupled interfaces between themselves. I use two main techniques to measure how teams are doing and whether further tweaks are necessary.

The first is to look at how the team measures success. Are the success criteria within a team aligned or in conflict with each other? If a team has been asked to solve a specific problem for all customers, but also has been asked to deliver a first version of that functionality to a customer that has been promised this in an RFP bid, there is an underlying conflict in satisfying this first customer on a specific timeline and to find a solution that will work great for all customers. Or the designer on the team may have been asked to improve the information architecture for the UI, but the team has also been asked to create an API that offers the functionality as a capability. In these cases, the team will be pulled in two different directions. Sometimes such goal conflicts can be difficult to detect because seemingly there is alignment. It is important to stay close to the teams and listen to what they struggle with. This will reveal the conflicts.

The second is what I call the “ping-pong analysis“: if you map communication between two individuals for each question or request (ping) and for each answer (pong), how many ping-pongs go quickly within the team. and how many ping-pongs go outside the team? And even more important: are there ping-pongs that really should happen frequently for the team to make progress, but where the time between a ping and a pong will be delayed? More than a day? Several days? More than a week? If you don’t have most ping-pongs within the team, you need to do something about competency, resources, empowerment, or in other ways try to remove dependencies on “pongs” that slow the team down.

If you as a leader are able to step out of making decisions and instead systematically enable your teams to make decisions and deliver value as described here, you are on the right track. If you are used to thinking that the most important decisions should be escalated and happen higher up in the organisation, you need to start thinking that they need to be done in the team or between the right teams, where the most knowledge is. Happy Conwaying!