Scaling Specialized Expertise

Modern software engineering is very enabling to an individual engineer, but with that enablement comes complexity. Likewise, even in areas we would consider “old news” the complexity of admistrating, operating, and interacting with those systems doesn’t go away.

When you’re dealing with optimizing that Postgres query, or why DNS doesn’t work you need specialized expertise: why this one thing, configured a naive way, isn’t working the way it should.

The broad expertise of even a ”T shaped engineer” doesn’t matter if what you need is the deep expertise, and often the deep part of an engineer’s area of expertise isn’t the part in question.

Now I’m the kind of engineer who loves to document things for the next person, especially when that next person may be future me. I also firmly believe in building up and making more productive the people around me.

So how can I help the organization learn and retain specialized information better? Especially in organizations that are in or post the specialization growth plane?

Enter Seeing Organizational Patterns

Seeing Organizational Patterns, my notes posits that an organization can be broken down by how it treats it’s workers on an almost Project Management Triangle of cooperation, control, and autonomy. That if you maximize one angel you have to let another one slide.

I further posit that the best way to scale specialized expertise depends on the organizational patterns used at that ‘shop.

Autonomy

Danger, Will Robinson

If an organization places premium on independent teams working on truly isolated functionality then some of these strategies may work best for scaling expertise.

Now be aware here: a highly collaborate or controlling organization can have autonomous parts for a time, but often wants to place those teams back in the fold when they “grow up”. Maybe an R&D project or a replatforming project, for example. In these cases I would go straight to the appropriate other section and maybe backtrack how to use those ideas in this one shot “special projects team”.

It’s also very possible the organization’s growth plane may also play a factor: before the specialization phase, where everyone is doing everything and things are very fluid, this may be the order of the day because even the teams are half-baked (“hmmm, we might need to split this team of 10 up, someday…“)

Strategies

Most of these work around building knowledge in the individual then sharing it across the team.

1. Individual amateur (in the “not trained” sense) or protégé (in the “pupil” sense) figuring out the hard thing and bringing it back to the team, either through code comments, some non-code documentation method, or some talk format (brown bag, lightning talk, etc) presented to the group:

* This also might be someone who just decided to go read the DNS book over nights and weekends!
* You might be able to spawn interest in cross-team but common engineering brown-bags. Everyone uses Java but is interested in Java 17, or clever Typescript tricks etc.
* Maybe even a regularly scheduled "guild" of interested Java (or React or General-Frontend or...) developers!

2. Training classes or conferences. Maybe one person goes, maybe it’s a team building exercise and everyone goes, but everyone takes their own notes and scaled expertise is learned that way: from the expert speaker to the listener. I do love going to a good conference as a team, years ago. But maybe it’s just someone going to the training and coming back with a “neat things I learned!” List. Which are fun and awesome!
3. The manager could realize the team really needs an expert of a certain kind and tries to hire one. This might be a T shaped individual where a person’s expertise area is the area in need, or it could be a specialist in only that one needed skill.
4. 1:1 pairing inside the team, with the subject matter expert, or everyone joins the subject matter expert while they work through an easy ticket, with the rest of the team asking questions as they do.
5. Lean into the Layer Cake. For example, ensure commits (or better yet, squash merges!) point to a ticket, whith then you may be able to back-reference a design document or some form of history.

You’ll notice some of these ideas are useful when nobody on the team knows something and it’s needed by the team so someone goes and learns it. Another thing, brought to mind by one of my reviewers, was when there’s a Subject Matter Expert on the team. In this case it’s very important for these Subject Matter Experts to share that specialized expertise, at least within the team (if not further)! This might be “the person who does the DevOps work on the team”, or “The person that knows the InterestCalculator module really well”.

Sharing the knowledge acquired could just be within that team, but I’m a huge fan of lunch and learn, show and tells, “lightning talk day”, or book clubs where you also invite friends of the team. They might only show up if a topic is of interest, but even in autonomous organizations there’s no cost to sending an invite. (Celebrate when anyone outside comes, because that’s not a given here).

Control

Control is one area where you have learning and knowledge-sharing that can go across multiple teams. This could be at a code level (“the department prefers JAX-RS style annotations vs Spring annotations”) or at a larger level (“We deploy to Kubernetes, always, and here’s a cluster for you that you’ll use”).

Important note here: sometimes control is knowing what not to control for. Someone wants to go off the set up, controlled golden path for something? Ok cool but that comes with disadvantage X, Y and Z. My favorite trick here is one well supported golden path, one less supported path of an entirely different technology, and beyond that you’re off-roading and unsupported. Assuming these two technologies are far enough apart from each other they can cover most of what you might want to do in the common cases.

Now sometimes it doesn’t matter if it’s control or cooperation, the same techniques can be used. But sometimes not.

Strategies

1. Abstract it away. If, for example, everyone uses the same CI/CD pipeline, and the contents of which are untouchable by the average bear, you don’t need to teach everyone how to write pipelines. And you can do that because the organization does not value everyone being able to have their own pipeline or whatever: you can give developers a half dozen knobs to turn and it’ll be enough to get their jobs done.
2. On demand complicated subsystem team. The team might be “that one person who knows SQL really well” to “that team that operates our Kubernetes clusters” to “the team that maintains CoreFunctionalityLib”. They have the specialized expertise, and going to an external resource is technically a way to scale that knowledge. (Personally I’m not convinced this is a great strategy, but it is one)
3. Principle Engineer / Architect mentoring and growing staff at scale individually. Like holding semi-optional weekly developer-only meetings across the org, and attempting to mentor team leaders under them
4. Use standards, patterns, starter packs, common libraries and standards to enforce org wide standards, and ways of thinking about a problem. A good starter pack can answer entire classes of questions (from “where do I put my code??!” To “ok, we’re expected to do input validation and here’s an example that shows me how”) and avoid classes of problems (nobody ever thinks about using global variables, because why would you - that’s a win).
5. Brown-bags are still useful here! Just maybe bigger! Instead of knowing the audience size will be just your team, and maybe a special guest will show up, instead it’s multiple teams showing up. You’ll need more than just 2 of those pizzas.

Finding and abstracting problems is my favorite play in situations like this: you don’t need to draw the team together for expensive training classes when you only have a couple knobs. One of the things Helm lets you do is teach the chart, not teach all Kubernetes to developers. Or a golden path of “we’ll do all of that for you by convention, but you can opt-out if you have to”.

Cooperation

Cooperation is control that everyone agrees on. Ok that’s not really the definition of that word, but maybe we can agree it is, for these purposes?

Both cooperation and control give you high level tools because of the similarities across teams. You can reduce cognitive load because you’ve abstracted something, lean on the fact that you have standards that are (mostly) all followed to automate common tasks and queries, or quickly onboard team members transferring from sibling teams.

But in addition to the strategies for “control” organizations, you have a couple more strategies in your hands for “cooperation” teams…

Strategies

1. Sandboxes and playgrounds. Can individuals over there quickly experiment with context and tools from over here to cooperate and collaborate quickly? Can they clone a repo with playground scripts, or open some kind of API response explorer to see what’s provided by a fellow team?
2. Scaled out cert programs: some kind of “cert-a-thon” where teams compete to get the most AWS certified team members or something. (Back to overlap with autonomous methods)
3. Keeping the Wiki. I <3 a good wiki, especially ones with yes good documentation of the fundamentals, but also lists of recorded videos that are applicable in certain situations (“onboarding”, “launching a new app”, “how we do incidents”, etc). Make sure each team has their own area for pieces they’re not yet ready - or don’t care to - share.
4. Principle Engineer / Architect mentoring Archetype is very important here too. Feedback needs to be addressed, processed, PRs merged, minds met etc. Now these’s an excellent talk on glue work, and how that needs to be (a senior engineer’s) explicit job (because they won’t code much anymore). Can you gently care for and herd the cats? Can the Principle Engineer lead through respect and not authority to get their job done?
5. Open RFC or technical plan process. Where anyone who wants to attend can come - or call - a review meeting. (My advice to engineers proposing a thing: for particularly contentious topics, have a pre-meeting with select advisors first so you know if your idea has merit or if you’ll get Tons O’ Pushback). These open meetings are a great way to scale the “why” behind a decision, which having that context is also part of specialized expertise.
6. Especially here an idea of a regularly meeting special interest / guild group is important, for peers to be able to scale information and feelings across teams, just amoung developers. Can foster cross-team colloration, expose areas where something’s not working for teams, etc. Sometimes an off-handed remark can be the spark to start something great!

Conclusion

Seeing Organizational Patterns is a great book, because you have a couple of different factions about how to run an organization:

1. The Daniel Pink way: people yearn for purpose, autonomy, and mastery . (I disagree with Pink on some stuff, but think this is somewhat true)
2. Teamwork (cooperation) makes the dream work, everyone’s voice matters, we’re a democracy after all (In my mind there’s a couple traps here, and having a good leader here avoids some of them)
3. ()Especially for things not directly related to the business domain): we should figure it out once then use that stamp all over the place (This sure sounds appealing, doesn’t it?)

And there’s autonomy, cooperation and control. Strategies that scale specialized expertise in one way may result in cultural pressure to conform (or not take off!) when the organization doesn’t value that pattern. Haskell’s a fun “let’s learn something random!” Lunch and Learn (and please do these, BTW!), but using Haskell in Prod will likely only work in an organization that places high value on autonomous teams.

Most of the time you don’t need to know exactly how this one weird thing works, but when you do you need to call in or find or become the expert…