Defining a Healthy Software Engineering Team

October 27, 2023

Walter Martens

Introduction

I was recently asked by a CTO to present what makes a healthy software engineering team and what are the primary roles of its leaders. Having been in every role on software development projects over the course of twenty years, I had a lot of opinions. As I organized my response for the CTO, the difficulty became narrowing the content down to a few key themes.

In this post, the first on my new site, I've tried to distill those themes into a quick and interesting read. I hope you'll find it valuable. I look forward to feedback.

TL;DR

To paraphrase Pascal and others, I have made this longer because I do not have the time to make it shorter. Here are the key takeaways:

The idea of a healthy team is a moving target. The best teams are constantly evolving and maturing.
A healthy engineering team is lean, talented, and has two-way trust with the stakeholders.
The primary responsibilities of leadership are to foster a culture of collaboration, set a vision, and enable continuous improvement.

A Note on Development Methodologies

I’m a strong believer in the benefits of an agile approach to software development for most scenarios. This post assumes an agile approach. I do not prescribe any particular flavor, but focus on the aspects of team development that underpin why methodologies like Scrum, Kanban, and XP prescribe the ceremonies and processes that they do.

The Healthy Team

I define a healthy team as one that is able to deliver consistent value to the organization over sustained periods. The lifecycle of any major project is a continuous search for equilibrium in a series of competing ideals; autonomy and accountability, agility and predictability, velocity and stability, among others. Each of these topics could be a post (or a book) of its own. The most successful teams are able to continuously mature and seek balance between these ideals.

That said, there are three key attributes that will help an engineering team deliver consistently.

Be lean - avoid talent dilution and communication overhead,
Talent - provide access to the knowledge and skills required, and
Trust - build strong trust relationships with stakeholders

Below, I've made a few points about what it means to be lean and talented, and the importance of trust and how to cultivate it.

Lean

In the constant effort to find balance between velocity and stability, a common reaction is to add more people. As Fred Brooks first described almost half a century ago in The Mythical Man Month, adding more people to a project doesn't equate to a linear increase in velocity. In fact, it often slows a project down, especially initially.

"Adding manpower to a late software project makes it later." - Fred Brooks

It is obviously easier to stay lean with a small team. However, there are situations where the complexity, scope, and ambition of a project requires a large team. For that reason, I wanted to add a quick note on scaling lean.

Scaling Lean

The more the team grows, the harder it is to avoid talent dilution and communication overhead. I often get asked, "then how do we scale?" The short answer is, "carefully and deliberately." There are numerous agile and semi-agile methodologies and frameworks, as well as specific architectures, that are intended to support scaling large projects. The larger the team, the greater the importance on scalable processes and consistent tooling, but all of these approaches have significant trade-offs and the bottom line is that new team members need to be carefully vetted for culture fit in addition to filling a required skillset.

Talented

This may seem overly obvious. What makes a great team? Talent! I don't think I've seen the 10X factor that is commonly attributed to great developers on a team, but there is certainly a wide-variance in talent levels and output from one developer to the next. Here are three concepts to keep in mind when building a talented team.

Having a talented team doesn't mean that the core team has expertise in every skill required for the project

Ideally, an autonomous team would have all the skills necessary to manage the entire DevSecOps ecosystem. They would build and manage deployment pipelines, automate testing, verify security, and design the user experience. Not to mention the flawless, self-documenting code they would be writing. For a obvious reasons, this lofty goal can be difficult, if not impossible, to achieve. This leads me to the next point. Don't hire for a specific skillset, hire strong technologists with the right mindset.

Hire smart and inquisitive critical-thinkers and give them the autonomy to succeed.

In an interview with Peter Siebel for his book, Coders at Work, Dan Ingalls (co-developer of Smalltalk) said, “If you grow up in a family where, when the cupboard door doesn’t close right, somebody opens it up and looks at the hinge and sees that a screw is loose and therefore it’s hanging this way vs. if they say, 'Oh, the door doesn’t work right; call somebody' - there’s a difference there.”

When I hire engineers, I'm looking for strong technical proficiency agnostic of the stack and I want the mindset of a person who lifts the hood when the engine makes a funny noise. Hire these people and give them access to learning resources and the space to fail fast. Encourage continuous learning, and provide access to external expertise through partnerships or consultants when necessary.

When augmenting a core team with external expertise, the external team needs to bring a value-add approach.

So, given that we have smart generalists on our team, there are going to be times when a specific required skill is missing. Whether it's devops, infosec, or a specific technology expertise, it's a common practice to consolidate this skill into a single team. The advantage of this shared-service approach that the centralized team can support many projects that don't each need full-time expertise. This model can work well when shared services have a clear value proposition and are not a bottleneck in the process.

I'll share a quick example of how I saw a shared-services team change from a bottleneck to a real value-add. This was a large program with dozens of teams and hundreds of developers. There was a dedicated devops team on the program and an infosec team shared across the larger organization. Initially, the devops team would just assign teams to individual experts and allow them to create the deployment pipelines for each team, working with infosec to ensure what was being deployed was within guidelines. As the program grew, the devops team became overwhelmed and infosec was often notified late of new pipelines or changes. Not only did this slow the project, but it added significant risk.

Working with an experienced devops leader, the team created a set of standard deployment pipeline templates with pre-approved infosec tooling and configuration. Delivery teams that were able to use the standard toolset, were able to expedite deployment via a self-service model. Now, the devops team was able to focus on just two priorities, keeping standard templates updated and servicing outlier teams that had special requests. By delivering value to the teams, they were able to scale to support the entire program with fewer resources.

Of course, this is just one example, but the point is that shared services need to be able to demonstrate value to the teams they support. This holds with the above principle of being lean and also helps to build trust between shared-services and the delivery teams they support.

Trusted and Trusting

By trusted team, I am not referring to security or access control. I'm referring to the trust relationship between the delivery team and the stakeholders. Trust is what empowers a team to take risks and innovate. It is what allows a team to be candid about current state and to ask for help when needed.

A strong working relationship across the broader team is critical to success. This includes the core delivery team, shared-services, product management, and any other leadership or stakeholders.

Delivery teams earn trust by deploying working software in frequent iterations and being candid about current state, and finding innovative solutions when tasks seem insurmountable.
As stated above, shared-services teams earn trust by providing value to the teams they support.
Product management earns trust by illustrating purpose, setting realistic priorities, and by providing timely feedback.
Leadership and other stakeholders can earn trust by setting a vision, staying engaged, and reacting to bad news with pragmatic adaptation.

When there is strong communication and a balance between accountability and autonomy, trust is built. When there is trust across the organization, the high-performing, healthy value.

Summary of the Healthy Team

A lean team and talented engineering team that establishes trust relationships will be able to deliver with high-value. The process an organization follows to get there is not universal, but the principles are. When cultivating this team, leadership plays a critical role. The second section of the post describes the role of leadership in more detail.

The Role of Leadership

This section assumes that hiring well is extremely important. I already stated above, hire smart and inquisitive critical-thinkers and give them the autonomy to succeed. Everything discussed here is predicated on hiring for the right mindset and culture fit.

Aside from hiring, the three areas of focus I want to call out for the role of leadership are:

Foster a culture of collaboration
Set the vision, and
Enable continuous improvement

Exactly how these are accomplished can vary tremendously based on several factors, including the size of the organization, the type of solution, and the maturity of the team.

Culture of Collaboration

Product, design, and engineering combine to make a three-legged stool upon which a successful solution is built. Too many times I've seen these teams siloed, when the goal should be co-delivery. As the organization grows, and the need for larger teams and shared services grows, more and more teams are typically created; architecture, quality, performance, devops, infrastructure, security, and support, to name a few. Each may become responsible for a slice of the pie.

Ideally, delivery teams and key stakeholders will be frequently (if not continuously) collocated. In a globally distributed team, where that is infeasible, regular communication can be facilitated through various ceremonies with regular cadence for questions and feedback.

Th goal of leadership is to find the balance between hours of meetings and too little communication. I tend to error on the side of too few ceremonies and encourage open door communication. On very large teams, it can become commonplace for developers to be in meetings for 4+ hours a day. For obvious reasons, this is less than ideal and speaks to the inefficiencies of large teams.

Strong leadership will seek to build a culture of collaboration by breaking down silos, encouraging open communication and creating a shared sense of ownership, regardless of team size.

Set a Vision

Ensure the team understands the motivation for the solution, not just the requirements.

Find balance between predictability and agility. When setting a vision, I like to stress why we are doing the project, what are the problems we're trying to solve, and how the solution will benefit the customer.

As the solution and the requirements evolve, the vision should stay relatively stable. Foster an environment that embraces change and sees it as a healthy part of the process. The vision needs to be constructed in a way to allow for this agility.

Enable Continuous Improvement

This is the part of the blog where I invalidate the premise. The question was posed to me as request for a definition of a healthy engineering team. However, it is not a binary matter of whether a team is healthy or not, but a continuum of maturity. Among leadership's primary responsibilities is to enable continuous improvement on this spectrum.

Continuous improvement applies to the product, process, and people. Encourage continued learning, especially in areas of relative weakness. Identify key goals of the organization and encourage development maturity through measurable goals. I've often used maturity matrices to define capabilities and track improvement within and across teams.

Summary

The question of what makes a healthy team is about the culture of the development organization and sustained productivity of the team(s) within it.

A talented and diverse team with the right balance of autonomy and accountability will be able to deliver value to the organization. The role of leadership is to foster a culture of collaboration, set the vision, and enable continuous improvement. In practice, this is a continuous journey of discovery and adaptation. The most productive teams I have worked with are constantly evolving and maturing and they endure the inevitable ups and downs of technology delivery by seeking innovation through collaboration.