architecture

A Practical Approach to Engineering Leadership

August 17, 2019 by Greger Wedel

In my previous post on the new engineering leadership skills needed in a devops world, I called for a more systematic approach to developing leadership capabilities in engineering. But I gave no answers. In this post, I’ll share my practical approach to developing leadership capabilities in an engineering organisation.

When developing great software, there are some fundamental questions all engineers have:

What can I decide?
Who can make decisions on things I cannot decide?
Who do I need to talk to or involve when resolving an issue?
Who determines the targets for what we make, how we make it, and how we go about doing it?
What is expected of me and what kind of rewards do I get when I meet expectations?

In a small startup team, you don’t really need a lot of these things explicit. The goals, actions, and rewards are mostly shared across the team, and everybody contributes where they see a need. The entire team has the same priorities, the tradeoffs are decided upon continuously, and changing course is done quickly.

However, you cannot grow much as an organisation without introducing slightly different priorities in different teams or roles. You are starting to introduce the cost of coordination. For a while, you can align everybody around the same priorities through some kind of coordination process or meeting that gives you the “top 5 things to work on this week”. But to allow rapid growth you need to at some point allow different people or teams to focus on different priorities. I have seen numerous approaches to solving the resulting coordination problem, and most either introduce too much process or too little.

There is no single answer that works for everybody in every situation. However, if we go to the fundamentals of how to build great software, there are four different areas that always have to be covered in some way:

Build high quality code, designs, and other tangible stuff
Knowing what to make to solve the customer’s problem
Knowing how to technically solve the problems and deliver something that will work over time
Knowing how to go about making the tangible stuff like code and actually delivered what’s agreed upon

The skills that go into each of these four areas are very different. And the more people involved, the more coordination and collaboration is needed to make sure everybody is pulling in the same direction. Ideally, you want everybody to be 100% productive, but that is not possible as long as there is some kind of dependency between what people are supposed to do. How do you get the productivity percentage as high as possible?

Typically, each of these responsibilities translates into one of four roles (not individuals):

Developers build stuff
Backlog owners manage the priorities and the definitions of what is to be made
Technical architects determine how to technically solve problems
Engineering managers lead people, the process of development, assign the right person to the right task, and ensure velocity

These are roles, not individuals, so many engineers will wear multiple hats. Also, there is one missing part here: the service or product. Each organisation has to break down its services and products (and thus code bases) and group them in a sensible way. Once that is done, it should be possible to identify the developers, backlog owner, the manager, and the architect for each. An engineer can be a developer on one code base/service, an architect on another, as well as manage a deliverable.

This break down into modules and components is absolutely critical to the productivity of your engineering organisation and subject to Conway’s law.

Unless you match the identified ownerships and roles with your modules and components and get that right relative to your organisational structure, you are introducing inefficiencies in your software architecture and how people communicate. This is a continuous process and needs to be adapted to the changing needs with respect to both features and the underlying system architecture.

Going back to the questions at the top of this post (that most people have in a software development organisation), it should now be possible to answer the questions. You don’t have to go through a central manager or (..!) a committee or a process, you can reach out directly to the person in charge who can make a decision. E.g. if you are a backlog owner and you need more engineers to work on your backlog, you can talk to a manager. If you need an evaluation for how to implement a certain functionality, you talk to the architect. If several components need to communicate through an API, you know exactly whom to invite to discuss this.

You thus get an organisation that can resolve most day to day matters without central coordination.

Typically, you will have triplets of lead people for each of your main components: a manager, an architect, and a backlog owner. The development team can thus be productive by getting help from each of these three. Ideally, all three roles/people are co-located physically for highest productivity since high velocity requires quick and frequent touch points.

There are two other roles missing in this picture: if the team is a product team, you need a product manager role (who could also be the backlog owner, dependent on skill set). If your component or product has a UI, you should also have a UX lead who can work with the backlog owner and product manager to get clarity on the deliverables. See also an earlier post on the backlog and the product owner role.

This model is also good for training and career development. There are four areas where skills can be built dependent on individual interest. As engineers grow from juniors to become more senior, they will take on more roles and expand what they take ownership of. Some may want to take on manager roles and lead teams. Some may want to deep dive in a very narrow technical area and become lead architect.

Remember that any changes in an organisation will take time. You need to repeatedly clarify the expectations, the opportunities, show that real authority to make decisions come with these roles, and systematically encourage people to take ownership. And if people are uncertain, you need to exemplify what concrete tasks and actions are necessary and expected.

The reward is an empowered engineering organisation that can deliver great software with high velocity!

A Production-Quality Flutter Starter App

July 18, 2019 by Greger Wedel

Production-ready flutter apps are rare species…

Flutter promises a lot. It has also quite a few new interesting characteristics that are attractive to a CTO. When is it production-ready for your next app and how does it stack up against native apps and React Native? Will it survive and thrive over time? And for what type of projects is it suitable already now?

When Flutter first came out of beta, I started playing with it and quickly achieved a lot and had great fun. But I couldn’t find much evidence and code that showed how to bring a large Flutter app into production and maintain it over time. As a CTO, I love playing with new technology, but I also want to understand maturity when it comes to flexibility, maintainability, support for features typically needed for large-scale production applications, and life-time cost vs other alternative eco-systems. I ended up pulling together my own Flutter starter app that includes some of the important production app features, just to see how to do it and what the challenges are. In this post, I present what I did. If the interest is big enough, I may cover specific features of the app in follow-up posts.

The resulting starter app can be found at https://github.com/gregertw/actingweb_firstapp. The README describes how to run it and start playing with it. It does not cover all the things I wanted to include (and that are needed in a production app), but it is a good start. Here’s a quick summary of what the app covers in a running starter app:

Separation of business logic in models and providers, and UI in a separate folder structure
Use of a global UI theme and custom icons for both iOS and Android
Use of scoped_model for app state management
Simple widget framework for handling logged-in, expired, and logged-out states
State management of login and login token using Auth0 including permanent storage for restarts
Localisation using i18n and the Android Studio/IntelliJ flutter i18n plugin to generate boilerplate
Testing using unit test framework and mocking
Use of a plugin that is not published (flutter_auth0)
Use of a OS native capability (location tracking) using a published plugin (geolocator)
Use of Firebase Analytics for usage tracking
Use of Firebase Crashlytics for crash reporting

First login screen
Redirect to Auth0 Universal Login
Location event stream

Obviously, different people value different things, so let me be explicit on what I was trying to achieve. First of all, separation of code related to UI and business logic makes it easier to get good feature velocity as developers can work on the UI and business logic independently. This is not enforced by Flutter, so you need to be careful when building up the app structure. I created a global UI theme (good), created a provider for authentication using a separate folder with code files for business logic for Auth0 authentication (also good), but copied code from geolocator’s example code where widgets and logic were in a single file (bad).

State management is super-important to avoid high complexity, errors in the UI, slow rendering of the app, and the need to re-write state code as the application evolves. I tried out setState, inheritedWidget, BLOC, Redux, and scoped model (overview of state management approaches here). Of these, I evaluated setState and inheritedWidget to be too simple for a larger project, while BLOC and Redux are both viable choices for applications with lots of data. BLOC fits better in with the Flutter language constructs, but if you are used to Redux, that is probably your preferred choice. For this not-so-data-heavy app, I chose scoped model as a simple, but very powerful state library that fits nicely into Flutter thinking.

There are lots of great posts on Flutter widgets and how to build up your app, but very few that shows how to pull together login screens, storing tokens (also permanently), and handling logged in/out state. A mobile app typically needs an access token to connect to backend APIs. I explored using Firebase as an authentication service, but decided to go for Auth0 as it was easier to show a more generic approach to handling logins and access tokens. Using shared_preferences, I could store the access token across app and device restarts.

There are two things everybody knows should be part of any project, but that you typically add too late: testing and localisation. I wanted to test basic unit testing, mocking, as well as UI widget testing. I played with http_server to set up a mock web server (code is still in the test/ folder), however, I managed to do what I wanted with mocking of Auth0. The UI widget testing is still missing (PRs welcome!), but I added simple app state testing, as well as mocking of login and logout.
Localisation is in its basic form a mess in Flutter, with lots of boilerplate. The only viable option I could find was to use an Android Studio/IntelliJ plugin called flutter_i18n. It uses .arb files with translations for each language and auto-generates the boilerplate necessary to inject a localisation class S into the widgets.

One of the concerns about Flutter is how the platform-independent overlay interacts with the iOS and Android platforms to make apps that can run on both platforms. I thus wanted to both use an OS-native capability in my app (device geo location), as well as test out how to integrate code into the app that needs OS-specific code. The flutter-auth0 plugin is not published and the project was somewhat dormant, but had what I needed for Auth0 authentication and authorisation functionality, so I forked the project and used that plugin for Auth0 support. Callbacks are needed for the authn/authz flows, and this is handled with OS-specific code. Also, the geolocator plugin uses OS-specific location code, but is a published and supported library/plugin, so I could then test both.

Finally, I wanted to add analytics and crash reporting capabilities, as well as test out how to integrate with one of the backend-as-a-service offerings. I explored AWS Amplify and Google Firebase, but from a library support, only Firebase had mature enough libraries, so Firebase realistically was the only option (about six months ago). Any production-grade app needs this support, and while I have not actually put this app into production, the integration was simple enough and with Google having bought Fabric, the Crashlytics part “came for free” (though with a bit too complicated configuration).

Overall, I came out with a positive view on the readiness for production for Flutter apps. Not all apps can easily be implemented in Flutter, but if your app does not have fancy and very custom UI requirements and only uses standard OS-level functionality, you can get a LOT of velocity by having a single code base and a single team. Investing in Flutter is still a bet on the future.

On the downside, you still work in two eco-systems in parallel. Upgrading Xcode/iOS, to new libraries (e.g. AndroidX), and build system changes requires an understanding of the eco-systems and preferably Swift/Kotlin(Java). You need to continuously update to the latest technologies and libraries to avoid that you get stuck in incompatibilities between components. The library/plugin system is powerful and you can lock the dependencies, which is a good thing, but you need to investigate the maturity and history of what you choose to dependent on. If not, you may end up stuck with an important component that is not actively maintained. This means that your team should have a decent level of iOS and Android eco-system expertise in addition to Flutter to make sure you navigate well.

Comparing with React Native, it is an eco-system that has more maturity (but less focused direction), but with similar dependencies on the underlying OS. Flutter has quickly become a viable alternative. At this point, the choice is probably less about technology and more about strategy, people, and skills. If you want your frontend developers to develop mobile apps and they are already using React for web, then React Native is a logical choice. If you see mobile apps more in the context of desktop apps/unified web/desktop development, if you have more business logic, and you want UI/UX to be shared across web, mobile apps, and desktop applications, your business logic developers may be more comfortable in a language like Dart that is very similar to Java.
Many technology choices are of such a nature that engineers tend to forget that often the biggest concern is people and competences, not technical capabilities!

There is still one big missing piece in my evaluation: deployment pipeline and lifecycle management. Since I haven’t put the app into production, I have not tested code signing, beta testing, A/B testing, as well as upgrading the app. There is very limited information to be found on these aspects. However, as with the evaluation on AWS Amplify, I suspect that the only real choice for now (unless you like pain) is to go for Firebase and the eco-system around Google. The same holds for app messaging, a feature that I would love to the app, but I haven’t had the chance to do.

Feel free to interact and help me improve the starter app at https://github.com/gregertw/actingweb_firstapp! I would also love feedback if you are using it for your projects and get interesting experiences!

Using Pivotal Assumptions to Increase Velocity

August 29, 2015 by Greger Wedel

Pivotal assumption – “An assumption on business case, priority, minimum viable offer, or requirement that if false will fundamentally change designs, deliverables, or timelines.”

In my post on how organisational factors increase the bad impact of architectural debt, I referred to how the lack of clear business objectives and priorities slow you down and prevent important architectural decisions from being made. The result is that you need to plan along two or more paths in parallel, which obviously slows you down. Sometimes this double work is needed while you are waiting for a decision, and if all outcomes are likely to happen. However, very often one outcome is more likely than the others, it is just difficult to get consensus or a decision has a big impact on something else, so people want to avoid a rash decision.

In these cases, you will get dramatically increased velocity if you can make an assumption about what the decision will be. Teams will get more focus, people will stop doing double work, some things can get lower priority, other things float up to the top. I call this a “pivotal assumption” because a lot hinges on such an assumption, and if you make the wrong assumption, you have lost some time (though often less than you think, because executing on two things in parallel is very inefficient). But what makes a pivotal assumption different from the actual decision?

First of all, you need to make the decision and options explicit and clearly defined, and get people’s acceptance that this is a decision that needs to be made, but that it may take time. You also need to identify the things that do not hinge on the assumption by explicitly stating that “x, y, and z are not impacted by this decision, so we will plan these separately.” You want to do it this way as this will explicitly decouple x,y, and z from your pending decision and will get rid of uncertainty: is x impacted by the decision??

Then, you need somebody in a position to postulate the pivotal assumption. Let’s say you are making a new product for the 10-100 employee segment, and product management is not sure whether import of users from ActiveDirectory is needed in the first version of the product. A senior architect, engineering manager, or senior product owner needs to postulate that “we should execute from here on based on the assumption that ActiveDirectory import will not be needed in this phase.” All discussions and work related to ActiveDirectory import should then be put on hold.

There is one final, crucial element in using pivotal assumption. You need to avoid that it actually becomes the decision by identifying the facts or evaluations that need to happen before a decision can be made and pro-actively follow up. Everybody will be acutely aware that the longer you execute based on the assumption, the more costly it will be to make a decision that is not the same as the assumption. This is an incentive to get a conclusion. Also, I recommend that you actively look for indications that the pivotal assumption may be wrong, and that when it becomes clear that the pivotal assumption was wrong, you swiftly change course and treat throw-away work as sunk cost.

If you can systematically identify areas where people have different assumptions when doing their work, you can make explicit pivotal assumptions to get temporary focus, and then quickly close the identified decisions. This will dramatically increase your velocity. You may think that you get a lot of throwaway work, but truth is that without pivotal assumptions, you get double work, which has the same cost, but in addition you get also the unnecessary code into your codebase, increasing complexity and cost of maintenance. The psychological effect of not having clarity also adds to the cost of executing on two tracks in parallel.

I call this method “hypothesis-driven development” as you continuously identify assumptions, you gather facts and try to falsify your hypothesis. The organisation can get pretty good at identifying good assumptions and quickly close them. Be warned though, unless you pro-actively gather facts to try to falsify your pivotal assumptions, you end up adopting the assumptions as your decisions.

The Changing Development Organisation: The Software Architect

July 20, 2015 by Greger Wedel

(intro to this post on devops)

It has been a busy few months. Since my last post I have transitioned to a new role as part of day to day development (see linkedin.com/in/greger for more details) in the Collaboration Infrastructure Technology Group (UC, telephony, telepresence, conferencing, etc) in Cisco. Earlier, as part of Office of the CTO, I influenced the medium to long-term technology and architectural strategy, but as I had limited involvement in the day to day operations, it was a challenge to develop architecture that bridged near-term goals with longer-term strategy. Now, I’m less involved in long-term technology strategy, but on a daily and weekly basis involved in designs, decisions, and priorities made as part of development. Quite refreshing and comes with a bunch of interesting challenges!

If you have followed my blog, you have probably understood that I see architecture as something tightly embedded in day to day development, but more than just the sum of tactical, short-term steps. Also, you have probably detected my desire to see product management team and architects partner and work on the same thing: identify the whats and hows of building great products and services. Agile has for a long time changed my organisation and the last year or so, devops processes have really started to impact how we work. We have hardware projects, traditional waterfall product releases, and cloud services. This means that in my technology group we are currently using various flavours of waterfall, agile, and devops processes in parallel. It puts a strain on coordination and especially challenges existing roles in the teams. This post is about some of these changes.

Classic product organisations used more or less waterfall processes, and many so-called agile development organisations are using agile terminology to execute on a mostly waterfall mindset. However, with an increased focus on devops processes, the combined product/service organisation is facing another set of challenges in combining everything from hardware products and their software loads through legacy, monolithic, hosted applications to devops microservices. Tying these together to deliver robust, scalable services that offer great user experiences is a challenge along many dimensions: culture, people, competence, programming languages, tools, project management, goal setting, measurements, management, and roles.

One particular challenge is that not only does what we do change, our roles have to change as well. This is easy in a small organisation, but in a big organisation, the expectations other people have to what you are supposed to do is very important to your ability to get things done. So, let’s say I’m used to architects writing specs, then I will likely continue to ask for specs. Other people may not think an architect should write specs anymore, and they will ask for other things and maybe ignore the specs. This creates frustration and tension.

I believe most people in the industry are starting to understand the fundamentals of how devops are changing the roles of the development team and the operations team, and that they basically need to become one integrated team (with some new roles and new definitions of existing roles). However, the role of the product manager is also changing in the process, as is the project manager’s role, the engineering director’s, as well as marketing, sales, support, and partners. They no longer have releases to plan around and need to change how they work. I may cover some of these other roles in later posts, but in this post I will concentrate on the architect’s role.

First of all, a few words on what an architect is. In my understanding, an architect is a developer, a member of the engineering organisation, with a special responsibility for understanding the short-term and long-term business and technical requirements and making sure that the right technical deliverable is delivered at the right time with the right quality. Sometimes this is to get it done and deliver something less ideal from a code point of view. Sometimes this is to tell the product owner that some labour-intensive refactoring needs to be done in order to deliver a feature. My personal view is that a good architect also truly understands the value of the product to the end users and customers, and that he or she is a partner to the product manager or owner in defining and evolving the product. An architect can be a senior member of a scrum team (in Cisco, these are called technical leaders), or a more dedicated role that covers multiple scrum teams (senior technical leaders or principal engineers). In larger organisations, system architects typically span multiple products or services to create functioning solutions. In my view, regardless of scope, to be effective, the architects should be embedded in the engineering organisation. Some architects lead development teams, others have a more standalone role. However, the most effective teams seem to always three roles covered when decisions are made: the product owner (representing the vision and the problems to be solved by the product), the architect (representing how to deliver with the technical integrity of the product in mind), and the development manager (representing the developers, and making the trade-off decisions).

The architect is thus a member of the product leadership team. And with agile and devops, this role becomes even more challenging with the need to engage in more areas. Here are the some of the main areas of challenges:

Software architecture and code structure: There are two big challenges I want to mention here. First of all, devops requires a modularisation into micro-services, and getting the right grouping of functionality and defining clean, scalable APIs are critical. Second, the whole thinking of how you develop a feature or a set of features need to change. Every time you commit code, you cannot break anything. This influences how you architect the feature, how you test it, and you need to think in terms of A/B testing, feature toggles, and metrics, metrics, metrics(!), and not in terms of feature branches and unit and integration testing.
Software development processes: An architect needs to engage in code repository management, branching strategy, build ande delivery pipeline, test automation, A/B testing, feature toggles, 360 reviews, and much, much more. The architect needs to police the process and make sure that scrum teams working on various micro-services have the same view on the process and don’t break the builds. Very few other people will have the code knowledge and competence required to identify where the issues are and address them. The architect becomes critical to making sure that the devops machinery keeps working.
Requirements and product management: A shift from waterfall to devops is a shift in power from product management to the engineering organisation. Where product management used to ask for features based on market knowledge and research, a high speed of introducing features, doing A/B testing, and measuring the impact and quality relegates all product managers (except those who are very technical) to focusing on the go to market aspects of the service, not the requirements. The architect’s role in this becomes critical as the architects are instrumental in defining what to measure, and what you measure, you improve (whether this power shift has to happen, is a separate topic…)
Project management: Traditional project (or program) management should totally disappear. The software development process, the build pipeline, the metrics, the scrum standups, the scrum of scrum meetings, kanban boards, and the definition and usage of the APIs across the services, together with the automated tests together define and drive the project. The traditional tools of a project manager, like Gantt charts, dependency tracking, weekly project status meetings, read-outs to executives, escalations, and decision meetings based on analysis, are of absolutely no use in a properly executed devops process (note that badly executed devops processes will give ample room for such old tools and processes…). The architect needs to engage in the overall execution of the devops project to ensure that there is a constant evolvement of proper architectural frameworks, guidelines, automation, gates, micro-service designs, API/protocol designs, and much more.
Architecture: Devops does not mean architecture work is not necessary. On the contrary, as the architecture is constantly and organically evolving, the architecture that emerges can either be a disaster or a robust construction. However, the “ivory tower” architect is history as the emerging architecture emerges both with and without a plan, and it can only be shaped by having constant touch with what is being developed. A spec or design document will be obsolete within a week.
Management: I see a lot of very competent, senior architects who are brilliant at designing a flow or a new protocol, but who are lacking in managerial skills. Running efficient meetings with action items, pro et contra evaluations, closing decisions, setting deadlines and targets, chasing deliverables, and driving consensus are just examples of typical managerial skills that becomes more important for an architect.

There is so much more to how devops is changing an architect’s work, as always, the devil is in the details, but these categories have been helpful for me in understanding some of the challenges in evolving architects towards “devops ready.” The shift is similar already with agile development processes, but becomes more visible with devops.

Do you believe there are other important things that impact the architect’s role in devops? I’m happy to hear from you!

My next post covers the importance of one shared backlog backlog with all the actual work items visible, as well as the product owner’s role. Not directly focused on devops only, but relevant to devops as a good backlog grooming process is essential to getting anywhere.

Distributed System Architectural Debt: protocol, layering, and flow debt

July 30, 2014 by Greger Wedel

This post is the second post on distributed system architectural debt. If you haven’t read the first post, I recommend that you read about why you acquire architectural debt and how feature-driven architecture development is a main contributor. I also there cover “abstraction debt”, the first of four types of architectural debt I cover in this series.

The second type of distributed system architectural debt is “protocol debt”. Without going into details on definitions, APIs expose an interface into a system, to operate on stored data, retrieve data, or trigger actions, i.e. an external system can do stuff remotely. However, a protocol defines a two-way flow of messaging between two or more entities in a system. When you design a protocol, you typically want to establish a messaging flow that can be reused, and while the content in the protocol can change, you want the protocol itself to be fairly generic and stable. Good protocol design takes time and experience, and often the value of the protocol is a factor of the number of systems using the protocol. We like to do protocol design in a standardisation body as we are able to validate the problem as worth solving with a protocol, we get a lot of experienced protocol designers to work on the problem, and the result is more likely to be a protocol that is widely used (if it becomes a standard). I have written earlier about standardisation and the challenges of working through standards bodies, and sometimes we must develop our own internal protocols, either in the hope of getting it standardised later or as a replacement for what could have been a standard.

In fact, the way we develop protocols internally is very similar to how it is done in standardisation bodies like IETF, through involvement of many engineers, iterations, and consensus (except that we also do joint development and integration testing early on). Thus, systematic use of protocols as a tool for reducing architectural debt has high costs and takes time. Sometimes, the problem must be solved through less labor-intensive approaches, either through a protocol defined by just a few of the engineers who should have been involved or by using a simpler API, or maybe more often, just reuse a protocol we have by overlaying meaning, for example by reusing a field in a protocol to mean something new. The result is ambiguity in meaning, more corner cases that can go wrong, and more states or scenarios that are undefined and open to interpretation. Also, the protocol gets more difficult to understand and implement. This debt is very difficult to get rid of, especially abusing an existing protocol, as it is rarely reversible. The problem also grows exponentially as the whole point of a protocol is to have it codified in a document with so much clarity that an engineer can implement based on the document and then do interoperability testing with other implementations to weed out bugs. Any document will have ambiguity, but when adding protocol debt, and thus more ambiguity, a protocol implementation will increasingly need the involvement from an engineer who has already done an implementation to get it right. Even then, new corner cases will be discovered and more testing is necessary for each new feature or implementation. The ability to improve the protocol through iterations is reduced. Using protocols defined in a standardisation body makes it harder to justify overlaying new meaning as we typically want the new meaning to be an acceptable addendum to the standard. With internally defined protocols, we have no such barriers.

Another type of protocol debt is insufficient protocol implementations. If you implement a SIP stack, there are a large number of RFCs to implement and then there are a number of things you need in a working SIP system that is not clearly specified in a standard. If you implement only the MUSTs, and you only test with only your own implementation, you get something that is compliant with the RFCs, but that doesn’t mean that you have implemented something that is interoperable. That may be fine if you are not really looking for interoperability outside your company and you have just one team working on a SIP stack. If you have multiple teams maintaining SIP stacks, then you have exactly the same problem as if you had a multi-vendor environment: you need interoperability. There are two approaches to get this: either you do end-to-end testing for all the use cases you need to support, or you basically do the same type of iterative interoperability testing done in the industry through events like SIPit. We do both, but while my preference is clearly on the latter, there are enough people who feel that generic interoperability testing is not necessary.

The third type of architectural debt, “layering debt”, is a multiplication factor on abstraction and protocol debt. Layering debt occurs when the same basic problem is solved by different people, but with different requirements or focus. Any good protocol design is based on layering, and layering is also useful in any large-scale system with multiple software components to split responsibilities for solving a certain task. However, there is no such thing as a “correct layering”, depending on what you are trying to achieve, a different type of layering may be the right choice. In video conferencing, scheduling a video meeting is a problem where layering is necessary. You have a calendar on your smartphone, you may have outlook with an exchange backend, you have a web site for scheduling meetings, you want to get a virtual conference room to meet in, you want to send out an invite with meeting details, and so on. The scheduling functionality can be solved through many different types of layering (or no layering at all), where each system in the architecture takes responsibility for handling one or more of the required functionality. When two different layering models are used to solve the problem (in our case we have e.g. webex-type scheduling and video conferencing-type scheduling), it is very hard to bring them together as the layers are not compatible. Typically, a problem is solved in one layer in one implementation and in a different layer in the other implementation. Or the same thing is solved in slightly different ways on the same layer. When we bring the two implementations together without a clean-up in layering, we get cross-layer, cross-implementation dependencies. Or what can be referred to as “duct-tape and piece of string architecture”.

Finally, the fourth type of architectural debt is “flow debt”. Layering debt will typically lead to flow debt as layering cross-dependencies are difficult to solve without introducing multi-step API calls and multi-hop protocol flows. When a certain feature or user experience relies on such multi-step flows, each step has a number of error situations that may occur and the probability of failure or undesired side-effects increases with each step added in the process. Flow debt is the mother of tangled, difficult-to-debug bugs that typically are discovered late in the release cycle and directly impacts timelines and release dates.

Being aware of these distributed architecture debts will allow you to manage and evolve your architecture by pro-actively establishing future architecture targets. These targets must have sound layering, the right abstraction level, good protocol craftsmanship, and carefully designed flows. However, there is a cost associated as you actually have to allocate engineering time to re-factor things that seemingly work fine (at the time) and each of the flavors need to evolve towards the target. Getting buy-in across the organisation to invest in distributed architectural debt reduction can be hard. You typically see an increased focus on “let’s just get things done” when distributed architectural debt grows. This organisational gearing effect is the topic of my next post in this series.