architecture

A Production-Quality Flutter Starter App

July 18, 2019 by Greger Wedel

Production-ready flutter apps are rare species…

Flutter promises a lot. It has also quite a few new interesting characteristics that are attractive to a CTO. When is it production-ready for your next app and how does it stack up against native apps and React Native? Will it survive and thrive over time? And for what type of projects is it suitable already now?

When Flutter first came out of beta, I started playing with it and quickly achieved a lot and had great fun. But I couldn’t find much evidence and code that showed how to bring a large Flutter app into production and maintain it over time. As a CTO, I love playing with new technology, but I also want to understand maturity when it comes to flexibility, maintainability, support for features typically needed for large-scale production applications, and life-time cost vs other alternative eco-systems. I ended up pulling together my own Flutter starter app that includes some of the important production app features, just to see how to do it and what the challenges are. In this post, I present what I did. If the interest is big enough, I may cover specific features of the app in follow-up posts.

The resulting starter app can be found at https://github.com/gregertw/actingweb_firstapp. The README describes how to run it and start playing with it. It does not cover all the things I wanted to include (and that are needed in a production app), but it is a good start. Here’s a quick summary of what the app covers in a running starter app:

Separation of business logic in models and providers, and UI in a separate folder structure
Use of a global UI theme and custom icons for both iOS and Android
Use of scoped_model for app state management
Simple widget framework for handling logged-in, expired, and logged-out states
State management of login and login token using Auth0 including permanent storage for restarts
Localisation using i18n and the Android Studio/IntelliJ flutter i18n plugin to generate boilerplate
Testing using unit test framework and mocking
Use of a plugin that is not published (flutter_auth0)
Use of a OS native capability (location tracking) using a published plugin (geolocator)
Use of Firebase Analytics for usage tracking
Use of Firebase Crashlytics for crash reporting

First login screen
Redirect to Auth0 Universal Login
Location event stream

Obviously, different people value different things, so let me be explicit on what I was trying to achieve. First of all, separation of code related to UI and business logic makes it easier to get good feature velocity as developers can work on the UI and business logic independently. This is not enforced by Flutter, so you need to be careful when building up the app structure. I created a global UI theme (good), created a provider for authentication using a separate folder with code files for business logic for Auth0 authentication (also good), but copied code from geolocator’s example code where widgets and logic were in a single file (bad).

State management is super-important to avoid high complexity, errors in the UI, slow rendering of the app, and the need to re-write state code as the application evolves. I tried out setState, inheritedWidget, BLOC, Redux, and scoped model (overview of state management approaches here). Of these, I evaluated setState and inheritedWidget to be too simple for a larger project, while BLOC and Redux are both viable choices for applications with lots of data. BLOC fits better in with the Flutter language constructs, but if you are used to Redux, that is probably your preferred choice. For this not-so-data-heavy app, I chose scoped model as a simple, but very powerful state library that fits nicely into Flutter thinking.

There are lots of great posts on Flutter widgets and how to build up your app, but very few that shows how to pull together login screens, storing tokens (also permanently), and handling logged in/out state. A mobile app typically needs an access token to connect to backend APIs. I explored using Firebase as an authentication service, but decided to go for Auth0 as it was easier to show a more generic approach to handling logins and access tokens. Using shared_preferences, I could store the access token across app and device restarts.

There are two things everybody knows should be part of any project, but that you typically add too late: testing and localisation. I wanted to test basic unit testing, mocking, as well as UI widget testing. I played with http_server to set up a mock web server (code is still in the test/ folder), however, I managed to do what I wanted with mocking of Auth0. The UI widget testing is still missing (PRs welcome!), but I added simple app state testing, as well as mocking of login and logout.
Localisation is in its basic form a mess in Flutter, with lots of boilerplate. The only viable option I could find was to use an Android Studio/IntelliJ plugin called flutter_i18n. It uses .arb files with translations for each language and auto-generates the boilerplate necessary to inject a localisation class S into the widgets.

One of the concerns about Flutter is how the platform-independent overlay interacts with the iOS and Android platforms to make apps that can run on both platforms. I thus wanted to both use an OS-native capability in my app (device geo location), as well as test out how to integrate code into the app that needs OS-specific code. The flutter-auth0 plugin is not published and the project was somewhat dormant, but had what I needed for Auth0 authentication and authorisation functionality, so I forked the project and used that plugin for Auth0 support. Callbacks are needed for the authn/authz flows, and this is handled with OS-specific code. Also, the geolocator plugin uses OS-specific location code, but is a published and supported library/plugin, so I could then test both.

Finally, I wanted to add analytics and crash reporting capabilities, as well as test out how to integrate with one of the backend-as-a-service offerings. I explored AWS Amplify and Google Firebase, but from a library support, only Firebase had mature enough libraries, so Firebase realistically was the only option (about six months ago). Any production-grade app needs this support, and while I have not actually put this app into production, the integration was simple enough and with Google having bought Fabric, the Crashlytics part “came for free” (though with a bit too complicated configuration).

Overall, I came out with a positive view on the readiness for production for Flutter apps. Not all apps can easily be implemented in Flutter, but if your app does not have fancy and very custom UI requirements and only uses standard OS-level functionality, you can get a LOT of velocity by having a single code base and a single team. Investing in Flutter is still a bet on the future.

On the downside, you still work in two eco-systems in parallel. Upgrading Xcode/iOS, to new libraries (e.g. AndroidX), and build system changes requires an understanding of the eco-systems and preferably Swift/Kotlin(Java). You need to continuously update to the latest technologies and libraries to avoid that you get stuck in incompatibilities between components. The library/plugin system is powerful and you can lock the dependencies, which is a good thing, but you need to investigate the maturity and history of what you choose to dependent on. If not, you may end up stuck with an important component that is not actively maintained. This means that your team should have a decent level of iOS and Android eco-system expertise in addition to Flutter to make sure you navigate well.

Comparing with React Native, it is an eco-system that has more maturity (but less focused direction), but with similar dependencies on the underlying OS. Flutter has quickly become a viable alternative. At this point, the choice is probably less about technology and more about strategy, people, and skills. If you want your frontend developers to develop mobile apps and they are already using React for web, then React Native is a logical choice. If you see mobile apps more in the context of desktop apps/unified web/desktop development, if you have more business logic, and you want UI/UX to be shared across web, mobile apps, and desktop applications, your business logic developers may be more comfortable in a language like Dart that is very similar to Java.
Many technology choices are of such a nature that engineers tend to forget that often the biggest concern is people and competences, not technical capabilities!

There is still one big missing piece in my evaluation: deployment pipeline and lifecycle management. Since I haven’t put the app into production, I have not tested code signing, beta testing, A/B testing, as well as upgrading the app. There is very limited information to be found on these aspects. However, as with the evaluation on AWS Amplify, I suspect that the only real choice for now (unless you like pain) is to go for Firebase and the eco-system around Google. The same holds for app messaging, a feature that I would love to the app, but I haven’t had the chance to do.

Feel free to interact and help me improve the starter app at https://github.com/gregertw/actingweb_firstapp! I would also love feedback if you are using it for your projects and get interesting experiences!

Using Pivotal Assumptions to Increase Velocity

August 29, 2015 by Greger Wedel

Pivotal assumption – “An assumption on business case, priority, minimum viable offer, or requirement that if false will fundamentally change designs, deliverables, or timelines.”

In my post on how organisational factors increase the bad impact of architectural debt, I referred to how the lack of clear business objectives and priorities slow you down and prevent important architectural decisions from being made. The result is that you need to plan along two or more paths in parallel, which obviously slows you down. Sometimes this double work is needed while you are waiting for a decision, and if all outcomes are likely to happen. However, very often one outcome is more likely than the others, it is just difficult to get consensus or a decision has a big impact on something else, so people want to avoid a rash decision.

In these cases, you will get dramatically increased velocity if you can make an assumption about what the decision will be. Teams will get more focus, people will stop doing double work, some things can get lower priority, other things float up to the top. I call this a “pivotal assumption” because a lot hinges on such an assumption, and if you make the wrong assumption, you have lost some time (though often less than you think, because executing on two things in parallel is very inefficient). But what makes a pivotal assumption different from the actual decision?

First of all, you need to make the decision and options explicit and clearly defined, and get people’s acceptance that this is a decision that needs to be made, but that it may take time. You also need to identify the things that do not hinge on the assumption by explicitly stating that “x, y, and z are not impacted by this decision, so we will plan these separately.” You want to do it this way as this will explicitly decouple x,y, and z from your pending decision and will get rid of uncertainty: is x impacted by the decision??

Then, you need somebody in a position to postulate the pivotal assumption. Let’s say you are making a new product for the 10-100 employee segment, and product management is not sure whether import of users from ActiveDirectory is needed in the first version of the product. A senior architect, engineering manager, or senior product owner needs to postulate that “we should execute from here on based on the assumption that ActiveDirectory import will not be needed in this phase.” All discussions and work related to ActiveDirectory import should then be put on hold.

There is one final, crucial element in using pivotal assumption. You need to avoid that it actually becomes the decision by identifying the facts or evaluations that need to happen before a decision can be made and pro-actively follow up. Everybody will be acutely aware that the longer you execute based on the assumption, the more costly it will be to make a decision that is not the same as the assumption. This is an incentive to get a conclusion. Also, I recommend that you actively look for indications that the pivotal assumption may be wrong, and that when it becomes clear that the pivotal assumption was wrong, you swiftly change course and treat throw-away work as sunk cost.

If you can systematically identify areas where people have different assumptions when doing their work, you can make explicit pivotal assumptions to get temporary focus, and then quickly close the identified decisions. This will dramatically increase your velocity. You may think that you get a lot of throwaway work, but truth is that without pivotal assumptions, you get double work, which has the same cost, but in addition you get also the unnecessary code into your codebase, increasing complexity and cost of maintenance. The psychological effect of not having clarity also adds to the cost of executing on two tracks in parallel.

I call this method “hypothesis-driven development” as you continuously identify assumptions, you gather facts and try to falsify your hypothesis. The organisation can get pretty good at identifying good assumptions and quickly close them. Be warned though, unless you pro-actively gather facts to try to falsify your pivotal assumptions, you end up adopting the assumptions as your decisions.

The Changing Development Organisation: The Software Architect

July 20, 2015 by Greger Wedel

(intro to this post on devops)

It has been a busy few months. Since my last post I have transitioned to a new role as part of day to day development (see linkedin.com/in/greger for more details) in the Collaboration Infrastructure Technology Group (UC, telephony, telepresence, conferencing, etc) in Cisco. Earlier, as part of Office of the CTO, I influenced the medium to long-term technology and architectural strategy, but as I had limited involvement in the day to day operations, it was a challenge to develop architecture that bridged near-term goals with longer-term strategy. Now, I’m less involved in long-term technology strategy, but on a daily and weekly basis involved in designs, decisions, and priorities made as part of development. Quite refreshing and comes with a bunch of interesting challenges!

If you have followed my blog, you have probably understood that I see architecture as something tightly embedded in day to day development, but more than just the sum of tactical, short-term steps. Also, you have probably detected my desire to see product management team and architects partner and work on the same thing: identify the whats and hows of building great products and services. Agile has for a long time changed my organisation and the last year or so, devops processes have really started to impact how we work. We have hardware projects, traditional waterfall product releases, and cloud services. This means that in my technology group we are currently using various flavours of waterfall, agile, and devops processes in parallel. It puts a strain on coordination and especially challenges existing roles in the teams. This post is about some of these changes.

Classic product organisations used more or less waterfall processes, and many so-called agile development organisations are using agile terminology to execute on a mostly waterfall mindset. However, with an increased focus on devops processes, the combined product/service organisation is facing another set of challenges in combining everything from hardware products and their software loads through legacy, monolithic, hosted applications to devops microservices. Tying these together to deliver robust, scalable services that offer great user experiences is a challenge along many dimensions: culture, people, competence, programming languages, tools, project management, goal setting, measurements, management, and roles.

One particular challenge is that not only does what we do change, our roles have to change as well. This is easy in a small organisation, but in a big organisation, the expectations other people have to what you are supposed to do is very important to your ability to get things done. So, let’s say I’m used to architects writing specs, then I will likely continue to ask for specs. Other people may not think an architect should write specs anymore, and they will ask for other things and maybe ignore the specs. This creates frustration and tension.

I believe most people in the industry are starting to understand the fundamentals of how devops are changing the roles of the development team and the operations team, and that they basically need to become one integrated team (with some new roles and new definitions of existing roles). However, the role of the product manager is also changing in the process, as is the project manager’s role, the engineering director’s, as well as marketing, sales, support, and partners. They no longer have releases to plan around and need to change how they work. I may cover some of these other roles in later posts, but in this post I will concentrate on the architect’s role.

First of all, a few words on what an architect is. In my understanding, an architect is a developer, a member of the engineering organisation, with a special responsibility for understanding the short-term and long-term business and technical requirements and making sure that the right technical deliverable is delivered at the right time with the right quality. Sometimes this is to get it done and deliver something less ideal from a code point of view. Sometimes this is to tell the product owner that some labour-intensive refactoring needs to be done in order to deliver a feature. My personal view is that a good architect also truly understands the value of the product to the end users and customers, and that he or she is a partner to the product manager or owner in defining and evolving the product. An architect can be a senior member of a scrum team (in Cisco, these are called technical leaders), or a more dedicated role that covers multiple scrum teams (senior technical leaders or principal engineers). In larger organisations, system architects typically span multiple products or services to create functioning solutions. In my view, regardless of scope, to be effective, the architects should be embedded in the engineering organisation. Some architects lead development teams, others have a more standalone role. However, the most effective teams seem to always three roles covered when decisions are made: the product owner (representing the vision and the problems to be solved by the product), the architect (representing how to deliver with the technical integrity of the product in mind), and the development manager (representing the developers, and making the trade-off decisions).

The architect is thus a member of the product leadership team. And with agile and devops, this role becomes even more challenging with the need to engage in more areas. Here are the some of the main areas of challenges:

Software architecture and code structure: There are two big challenges I want to mention here. First of all, devops requires a modularisation into micro-services, and getting the right grouping of functionality and defining clean, scalable APIs are critical. Second, the whole thinking of how you develop a feature or a set of features need to change. Every time you commit code, you cannot break anything. This influences how you architect the feature, how you test it, and you need to think in terms of A/B testing, feature toggles, and metrics, metrics, metrics(!), and not in terms of feature branches and unit and integration testing.
Software development processes: An architect needs to engage in code repository management, branching strategy, build ande delivery pipeline, test automation, A/B testing, feature toggles, 360 reviews, and much, much more. The architect needs to police the process and make sure that scrum teams working on various micro-services have the same view on the process and don’t break the builds. Very few other people will have the code knowledge and competence required to identify where the issues are and address them. The architect becomes critical to making sure that the devops machinery keeps working.
Requirements and product management: A shift from waterfall to devops is a shift in power from product management to the engineering organisation. Where product management used to ask for features based on market knowledge and research, a high speed of introducing features, doing A/B testing, and measuring the impact and quality relegates all product managers (except those who are very technical) to focusing on the go to market aspects of the service, not the requirements. The architect’s role in this becomes critical as the architects are instrumental in defining what to measure, and what you measure, you improve (whether this power shift has to happen, is a separate topic…)
Project management: Traditional project (or program) management should totally disappear. The software development process, the build pipeline, the metrics, the scrum standups, the scrum of scrum meetings, kanban boards, and the definition and usage of the APIs across the services, together with the automated tests together define and drive the project. The traditional tools of a project manager, like Gantt charts, dependency tracking, weekly project status meetings, read-outs to executives, escalations, and decision meetings based on analysis, are of absolutely no use in a properly executed devops process (note that badly executed devops processes will give ample room for such old tools and processes…). The architect needs to engage in the overall execution of the devops project to ensure that there is a constant evolvement of proper architectural frameworks, guidelines, automation, gates, micro-service designs, API/protocol designs, and much more.
Architecture: Devops does not mean architecture work is not necessary. On the contrary, as the architecture is constantly and organically evolving, the architecture that emerges can either be a disaster or a robust construction. However, the “ivory tower” architect is history as the emerging architecture emerges both with and without a plan, and it can only be shaped by having constant touch with what is being developed. A spec or design document will be obsolete within a week.
Management: I see a lot of very competent, senior architects who are brilliant at designing a flow or a new protocol, but who are lacking in managerial skills. Running efficient meetings with action items, pro et contra evaluations, closing decisions, setting deadlines and targets, chasing deliverables, and driving consensus are just examples of typical managerial skills that becomes more important for an architect.

There is so much more to how devops is changing an architect’s work, as always, the devil is in the details, but these categories have been helpful for me in understanding some of the challenges in evolving architects towards “devops ready.” The shift is similar already with agile development processes, but becomes more visible with devops.

Do you believe there are other important things that impact the architect’s role in devops? I’m happy to hear from you!

My next post covers the importance of one shared backlog backlog with all the actual work items visible, as well as the product owner’s role. Not directly focused on devops only, but relevant to devops as a good backlog grooming process is essential to getting anywhere.

Distributed System Architectural Debt: protocol, layering, and flow debt

July 30, 2014 by Greger Wedel

This post is the second post on distributed system architectural debt. If you haven’t read the first post, I recommend that you read about why you acquire architectural debt and how feature-driven architecture development is a main contributor. I also there cover “abstraction debt”, the first of four types of architectural debt I cover in this series.

The second type of distributed system architectural debt is “protocol debt”. Without going into details on definitions, APIs expose an interface into a system, to operate on stored data, retrieve data, or trigger actions, i.e. an external system can do stuff remotely. However, a protocol defines a two-way flow of messaging between two or more entities in a system. When you design a protocol, you typically want to establish a messaging flow that can be reused, and while the content in the protocol can change, you want the protocol itself to be fairly generic and stable. Good protocol design takes time and experience, and often the value of the protocol is a factor of the number of systems using the protocol. We like to do protocol design in a standardisation body as we are able to validate the problem as worth solving with a protocol, we get a lot of experienced protocol designers to work on the problem, and the result is more likely to be a protocol that is widely used (if it becomes a standard). I have written earlier about standardisation and the challenges of working through standards bodies, and sometimes we must develop our own internal protocols, either in the hope of getting it standardised later or as a replacement for what could have been a standard.

In fact, the way we develop protocols internally is very similar to how it is done in standardisation bodies like IETF, through involvement of many engineers, iterations, and consensus (except that we also do joint development and integration testing early on). Thus, systematic use of protocols as a tool for reducing architectural debt has high costs and takes time. Sometimes, the problem must be solved through less labor-intensive approaches, either through a protocol defined by just a few of the engineers who should have been involved or by using a simpler API, or maybe more often, just reuse a protocol we have by overlaying meaning, for example by reusing a field in a protocol to mean something new. The result is ambiguity in meaning, more corner cases that can go wrong, and more states or scenarios that are undefined and open to interpretation. Also, the protocol gets more difficult to understand and implement. This debt is very difficult to get rid of, especially abusing an existing protocol, as it is rarely reversible. The problem also grows exponentially as the whole point of a protocol is to have it codified in a document with so much clarity that an engineer can implement based on the document and then do interoperability testing with other implementations to weed out bugs. Any document will have ambiguity, but when adding protocol debt, and thus more ambiguity, a protocol implementation will increasingly need the involvement from an engineer who has already done an implementation to get it right. Even then, new corner cases will be discovered and more testing is necessary for each new feature or implementation. The ability to improve the protocol through iterations is reduced. Using protocols defined in a standardisation body makes it harder to justify overlaying new meaning as we typically want the new meaning to be an acceptable addendum to the standard. With internally defined protocols, we have no such barriers.

Another type of protocol debt is insufficient protocol implementations. If you implement a SIP stack, there are a large number of RFCs to implement and then there are a number of things you need in a working SIP system that is not clearly specified in a standard. If you implement only the MUSTs, and you only test with only your own implementation, you get something that is compliant with the RFCs, but that doesn’t mean that you have implemented something that is interoperable. That may be fine if you are not really looking for interoperability outside your company and you have just one team working on a SIP stack. If you have multiple teams maintaining SIP stacks, then you have exactly the same problem as if you had a multi-vendor environment: you need interoperability. There are two approaches to get this: either you do end-to-end testing for all the use cases you need to support, or you basically do the same type of iterative interoperability testing done in the industry through events like SIPit. We do both, but while my preference is clearly on the latter, there are enough people who feel that generic interoperability testing is not necessary.

The third type of architectural debt, “layering debt”, is a multiplication factor on abstraction and protocol debt. Layering debt occurs when the same basic problem is solved by different people, but with different requirements or focus. Any good protocol design is based on layering, and layering is also useful in any large-scale system with multiple software components to split responsibilities for solving a certain task. However, there is no such thing as a “correct layering”, depending on what you are trying to achieve, a different type of layering may be the right choice. In video conferencing, scheduling a video meeting is a problem where layering is necessary. You have a calendar on your smartphone, you may have outlook with an exchange backend, you have a web site for scheduling meetings, you want to get a virtual conference room to meet in, you want to send out an invite with meeting details, and so on. The scheduling functionality can be solved through many different types of layering (or no layering at all), where each system in the architecture takes responsibility for handling one or more of the required functionality. When two different layering models are used to solve the problem (in our case we have e.g. webex-type scheduling and video conferencing-type scheduling), it is very hard to bring them together as the layers are not compatible. Typically, a problem is solved in one layer in one implementation and in a different layer in the other implementation. Or the same thing is solved in slightly different ways on the same layer. When we bring the two implementations together without a clean-up in layering, we get cross-layer, cross-implementation dependencies. Or what can be referred to as “duct-tape and piece of string architecture”.

Finally, the fourth type of architectural debt is “flow debt”. Layering debt will typically lead to flow debt as layering cross-dependencies are difficult to solve without introducing multi-step API calls and multi-hop protocol flows. When a certain feature or user experience relies on such multi-step flows, each step has a number of error situations that may occur and the probability of failure or undesired side-effects increases with each step added in the process. Flow debt is the mother of tangled, difficult-to-debug bugs that typically are discovered late in the release cycle and directly impacts timelines and release dates.

Being aware of these distributed architecture debts will allow you to manage and evolve your architecture by pro-actively establishing future architecture targets. These targets must have sound layering, the right abstraction level, good protocol craftsmanship, and carefully designed flows. However, there is a cost associated as you actually have to allocate engineering time to re-factor things that seemingly work fine (at the time) and each of the flavors need to evolve towards the target. Getting buy-in across the organisation to invest in distributed architectural debt reduction can be hard. You typically see an increased focus on “let’s just get things done” when distributed architectural debt grows. This organisational gearing effect is the topic of my next post in this series.

Distributed System Architectural Debt: The mother of all debts! And it’s feeded by features…

July 12, 2014 by Greger Wedel

In my previous post, I asked the question “Can you get 4,000+ engineers to work together?” The answer is, in the traditional “together sense”, an obvious no! However, just like ants are building an ant hill, not by talking to each other, but by each ant following a set of rules or behaviours specific to their role and by using simple signals, you can get way more than 4,000 engineers to work together by having architectural principles and rules that everybody follows. In the classic waterfall engineering organisation, such rules are codified in documents: technical specifications, solution requirement documents, architectural design documents and so on. Once the overall design is agreed, implementation can be done. In an agile product organisation, there are many people longing back to the times when such documents were used. Truth is that architecture by specs never really worked that well unless the problem to solve was really well understood. Preferably you had a problem that had been solved before, so each iteration would improve the architecture and process. If the process of implementing the specification itself gave significant new understanding to the problem and how to solve it, then the specification had to be changed and big changes gave big impacts to timelines, scopes, and how to implement.

You never get things right the first time, so improvement through iterations is key. If every problem is solved as if it is new, then there is little reuse of understanding from earlier problems you have solved. Thus, the key to improvement and great products and services is to identify patterns or operations that are repeatable, make these explicit, and then improve them through iterations. In a waterfall organisation, the process itself and the specifications become something that is repeatable (when for example you build a car), but in an agile software organisation with a high degree of change, with no elaborate specifications and with a minimum of process, other sets of repeatable patterns or operations must be found.

However, identifying what is repeatable is not enough. You must isolate what is repeatable as well. If not, you are not able to improve as everything must change in sync in every iteration. You isolate what is repeatable through abstractions. Abstractions can be specified through layering, APIs, protocols, modules, services, and so on. All should have clean and improvable interfaces and implementations. These abstractions are absolutely key to the ability to iterate and improve. The challenge in an agile organisation is that you typically build out the architecture as you iterate, i.e. you don’t start out with a clean, abstracted architectural design. So, as you evolve, you need a lot of refactoring and tight management of the overall system architecture. If you don’t invest enough in managing your architecture, you end up with a tangled mess instead. In a large organisation doing agile, you would be wise to organise your internal software development more similar to how the Internet has been developed: through independent teams developing based on a set of standards codified by organisations like W3C and IETF.

Going back to the product-level: technical debt, and design debt within individual products and services are increasingly better understood and managed. In agile, debt is systematically reduced as part of the development process. However, it is still difficult to get product management and management to understand this aspect of software development. Particularly true for the product marketing community (as opposed to product managers) and MBA-managers as they have limited engineering background. Engineers know that not all your resources can be spent on new features, but I often see them hide re-factoring and technical debt reduction in conservative project estimates instead of showing management the real costs of evolving and maintaining a code base for short-term and long-term health.

My problem, working on the architecture level across a 4,000+ engineering organisation, is that we also add debt outside the products and services we build: in how we make them work together. This is what I call distributed system architectural debt, or just architectural debt for short. Understanding, planning, and executing on managing this kind of debt is extremely difficult, and even more so when there is limited understanding of technical and design debt within each product organisation. It tends to add up…

We (Collaboration Technology Group in Cisco) have 4,000+ engineers who use a mix of waterfall and various flavours of agile (and we do both hardware, software, and services), we release our products with various cadence and thus often not in sync, and in order to make sure we move forward, each team needs to be able to plan, execute, and control their own projects. How the products interact evolves over time, we add new features and functionality all the time, and there is no upfront understanding of a “right way” products should interact and work with each other. This is necessary to foster innovation and get the speed to market of new features. However, our customers expect everything to work together in all the different ways they can work together (and sometimes other ways as well), an impossible task as we cannot test every permutation.

So, in order to get feature velocity, we need to allow the same technical problem to be solved in slightly different ways in different parts of the organisation. Absolute coordination requirements would absolutely kill any progress. However, once we have three, four, or maybe even five different ways of doing the same thing, the number of combinations of what should work is growing, and where we should have one test that would test a generic functionality (common to all the five different ways), we have let’s say 5 x 3 = 15 different tests to run if each of the five approaches to the same thing interacts with three different other things. So, if we don’t want to slow down feature velocity by requiring upfront coordination, we need a way of finding the “best approach wins” and a migration towards a single, shared method. This is just in theory, because each of these teams tried to solved a slightly different problem, had limited time to do proper generalisation and design, and ended up with a design solution that does not really fit the other teams’ needs. Voila, we are stuck in “no best approach exists”, and five different development trajectories with product management asking for new features on top of a design that was shaky in the beginning. I call this feature-driven architecture. The consequence is of course that the feature velocity we wanted in the first place is now slowed down, not by coordination, but by testing and because it gets increasingly difficult to make everything tick.

A typical large organisation response to the problem of multiple designs and multiple products with overlap is to synchronise and agree to a shared approach through a warped, time-consuming, regression-to-the-mean consensus process. In the absence of clear priorities, everything is equally important and everybody has an equal position at the table. The relative priority between something important to an existing revenue stream vs another important thing to a new product or service is typically not defined. There is also a lack of technical people who understand more than their own problem area, and there is a lack of strong, technical people with an authority in the organisation to actually make a technical decision that has important business impacts. And as architectural decisions often have clear business impacts, product management needs to be consulted as well. There is a lack of strong, technical product managers, so they focus on features for their product or service and have little to contribute on the fundamental architectural choices that have big business impacts later on, potentially across all the products and services.

Thus, 10, 20, 30 or even more people are brought together in large meetings to make sure all stakeholders are involved. Typically, engineers/architects work in groups, product managers work in groups, and some of them come together in either face-to-face workshops or large sync meetings. Eventually (if important enough), they prepare a consensus-driven read-out to directors and VPs. Since consensus was the goal, no real options have been identified and no pro et contra evaluations have been done, and the result is a mixed bag of something that nobody is really satisfied with. The directors and VPs grow impatient with the lack of progress and finally make a decision far outside their competence area and with limited insights into the impacts of their decision. This will haunt them later, and they will change the decision later down the road.

Am I exaggerating? Well, a bit, but truth is that any clean, easy to execute decision will have somebody on the loosing side, somebody who didn’t get their product or their design chosen as the path forward. Making that decision requires leadership and guts, but it is often too easy to avoid those tough calls. Knowing when consensus is called for and when a trade-off must be made, is one of the most important technical leadership abilities in a large organisation. However, the traditional approach with “one, strong technical chief” who will make the calls does not work as the number of evaluations to be done are too many for a single person to address. The only way to scale is to set clear business objectives and corresponding architectural plans to allow experts locally in each team to make the right decisions.

But, if you don’t get clear plans and trade-offs, what are you left with? Well, we will have to design for the consensus we achieved, both on the business objectives and architecture. This is always too big to be addressed in the next development cycle. Time is short, so let’s focus on what we need out the door the next 6-12 months! Okay! Back to feature-driven architecture!

You may say, ask product management, they should know where you are going! Well, as mentioned above, a typical product manager (not the one we wish we had) will go: “in phase 1, we need features a,b,c, and d, but we need to design for phase 2, where we will need …” and then they add everything they think they might need. Unless there is very clear guidance from upper management on what NOT to do, product management will hedge their asks and ask for pretty much everything they can think of. The result is that the architects have to prioritise which architectural problems to solve with very limited guidance. But we cannot do everything, so again, how do we prioritise? The answer is very often that we choose the stuff that we need to get done to get features in phase 1 out the door and then we do a little bit on some of the stuff we think may be important in phase 2.

Allowing features to drive design, not the generic architectural problem to be solved, introduces “distributed system architectural debt”. The debt that most directly comes as a result of feature-driven architecture, is “abstraction debt”. Getting the right level of abstraction is very difficult when you design APIs. Too little abstraction and your API only allows “set PIN code for this type of conference for this type of user when it is scheduled”. Too much abstraction and your API is a limited and meaningless interface “set variable Y with value X”. Both are responses to feature-driven architecture as the wider context of the problem to solve is not known, only the feature. The problem of setting the right level of abstraction is that you really need to understand the generic problem you are trying to solve. When you are working agile and you are feature-driven, you may not know how your API will be used later. Using a particular methodology, like REST, forces a certain design (e.g. identification of resources and exposure of data for each). This enforcement is of help because you have implicitly done the systematic groundwork for a good API. However, REST (or other approaches) is not the best choice for all APIs (in particular for hardware products), and you rarely have the luxury of ignoring all the existing APIs already implemented in your portfolio (unless you are a start-up or you are allowed to start from scratch).

In my next post, I will deep dive more into the consequences of feature-driven architecture and cover the three other types of distributed architectural debt I have identified: protocol debt, layering debt, and flow debt.