Organizing at the Edge of Chaos
Early in the presidential contest between the newly-presumptive Democratic nominee Barack Obama and already named Republican John McCain, Elisabeth Bumiller published “Cast of 300 Advises Obama on Foreign Policy” to which the New York Times gave page-one real estate. Largely an insightful “who’s-who” of his advisers for this delicate area of national concern, the article also offers remarkable portraits of the stark contrast between the organizing styles of two the candidates. One was simple, the other complex. Guess who won.
Bumiller describes Obama’s apparatus as a “huge 300-person foreign policy campaign bureaucracy organized like a mini State Department to assist a candidate whose limited national security experience remains a concern to many voters.” The article goes on to describe an “infrastructure” of twenty teams that focus on regions (e.g., Asia and China) and issues (e.g., nuclear proliferation). By contrast, Bumiller describes John McCain’s approach as “a far smaller and looser foreign policy advisory operation, about 75 people in all, and none are organized into teams.”
So there you have it. McCain had direct input from 75 people, an incredibly large “span” of “direct reports” by any organizational measure and an extremely flat organization. No “middle” men or women. Perfect for simplicity and direct communication. McCain, the ostensibly more experienced of the two men in foreign policy, didn’t think he needed nuanced views regarding foreign policy, and he didn’t get many.
Obama by contrast had a core team of six people managing twenty separate pipelines of progressively synthesized input from advisors with great depth of expertise. Larger teams had their own sub-teams, such as the forty-person nuclear proliferation team organized into eight working groups by Brookings Institution and National Security Council veteran Ivo Daalder.
In network terms, members of these teams are themselves hubs of networks. Thus, each of Candidate Obama’s 300 advisors served as a funnel for a torrent of highly nuanced expertise flowing his way.
Compared with McCain, Obama had a smaller number of people reporting directly to him on foreign policy but, and it’s a very important but, they connected to an extremely complex organization with many trusted middle men and women. Perfect for making complex decisions. Worth noting also is that this complex hierarchical structure was able to function and learn at the warp-speed required of a presidential campaign.
One subtle message in President Obama’s 2009 Inaugural Address is that all of us must face complexity unflinchingly. Complexity is on its own growth curve. We can’t stop it, but we can change how we see, understand, adapt to, and employ it. This is a major reason why transparency is so important. For people at all levels to lead effectively and make choices that are informed and wise, both locally and globally, we have to be able to see the whole of what we’re doing in all its swirling, knotted, twisted, gummed up complexity.
Is Flatter Better?
In this season of reorganization (each layoff, merger, and acquisition precipitates it), how are you facing complexity? Are you reducing or increasing your ability to make good decisions?
For the past thirty years or so, the prevailing wisdom about organizations is this: the flatter, the better. An inch-high and a mile wide. Smash the hierarchy. Nowhere was this more evident than in the corporate press release of the then-new CEO of BP. In October, 2007, Tony Hayward said his company was determined “to improve performance by simplifying how the company is structured and run.” While emphasizing that they have the right strategy and resources, he described BP’s problem this way: “…we are not consistent and our organization has grown too complex.”
To remedy the situation, BP planed to adopt more standardized procedures and reduce the number of management layers from 11 to seven.”D What major benefit did Hayward expect to gain from redesigning the organization? “… [T]he revenue boost expected from greatly improved operational efficiency over the longer term.”
No one would argue that simplification is indeed more efficient, but here’s the rub: It’s not necessarily more effective.
On the broader point of the benefits of reorganization, we agree: Better organizational design offers enormous competitive advantage. Organization, after all, leverages all other advantages. But did BP engage this challenge with the right frame of mind? Certainly, the study Hayward commissioned that identified “7,500 ‘operational interfaces’ – that is, potential management blockages”4 was on the right track. Gaining organizational benefits of the type BP’s chief desires will not be easy for his or for any other enterprise. Nor is it easy to remove four levels in an organization of roughly 100,000 people. Based on our research, it might even be suicidal. It is highly unlikely BP reached its goal.
Dogmatic global mandates, like one that says an organization must have no more than seven levels or that all managers should have ten reports (which a global financial management firm facing layoffs just executed), ignore other realities of business life. The number of levels your organization needs, or the optimal reporting span of your leaders, our research shows, is likely a function of what those units are actually doing.
Extensive study of one organization’s structure shows that some parts of organizations are shallow, others deep—depending on what they’re doing. Groups whose primary need is to communicate call for shallow structures that allow them to quickly spread messages; units engaged in complex decision-making require deeper structures that accommodate more specialization. The best structure fits the work at hand.
Our conclusions come from a detailed investigation of “Eleum,” our pseudonum for one highly complex organization, the key 5000-employee unit of a global enterprise similar to BP, along with a variety of smaller pilot studies that we’ve conduced. Likewise, our findings are consistent with our preliminary analysis of the senior levels of the U.S. federal government inherited by the Obama Administration (See NetAge Report #2, Analysing the Organization as a Network).
Over a four-year period, we analyzed Eleum’s formal hierarchy of who-reports-to-whom. We mapped its organization chart as a simple network of nodes and links, drawing our data directly from its HR system. Looking at what we found, we developed this hypothesis:
An organization seeks to minimize communication pathways while maximizing its capacity for complex decision-making.
Organizations are islands of relative order in seas of chaos that stretch far beyond local horizons. Suddenly storms erupt—thanks to revolutionary technologies, unforeseen competitors, safety accidents, world events—that bring chaotic change. Such storms are gaining strength and blowing in more often. In response, enterprises find the old order shaken up and in urgent need of reorganization. Comes then the time, as it has with the economic tumult of 2008, the need for all of us to decide how to rebalance order and chaos.
Our purpose is to help you decide whether flatter is indeed always better for organizations. Or, do some circumstances call for deeper, more complex organizations? If so, how do you decide?
We derive principles from looking at hierarchical structure in detail, so you can apply them to daunting challenges we all face, both inside and outside government. The same analysis and principles help small organizations inside and outside big organizations optimize their own structures to fit their local situations.
Between Order and Chaos
Networks ripple. A small decision here plays out as major activity elsewhere in the web of people and positions. Big effects arise from many small movements. Abstractions at a large scale become everyday local juggling acts for managers and staff across the hierarchy.
Organizations need order and stability, flexibility and creativity (see Figure 1). The structure must provide sufficient constraints to maintain integrity and enough freedom to innovate and adapt. Sufficient sameness and commonality have to mix with requisite variety and difference. Otherwise, the organization is either completely moribund or a total madhouse.
Executives struggle to manage these contrasting forces. They find themselves simultaneously bringing some things to center and pushing other things out, simplifying in some places, “complexifying” (if that were a word) in other places. They push for more collaboration over here (perhaps to better deliver services) and more competition over there (perhaps to control costs).
From the early dawn of the field of information sciences, its thinkers like Claude Shannon, Norbert Weiner, and Nobelist Herbert Simon have regarded organizations as “communication systems.” At the same time, the first generation of systems scientists (including Ludwig von Bertalanffy, Kenneth Boulding, and James Grier Miller) considered organizations prime examples of complex systems—living systems—like cells and organisms.
Today’s cross-disciplinary scientists call organizations “complex adaptive systems.” These whole systems comprise interrelated parts that are also adaptive entities, ones that are able to learn from experience, to change and to evolve. Emergence is what happens when they generate new levels of order, a property essential to complex systems. John Holland frames this elegant process simply in the title of his book, Emergence: From Chaos To Order.
As complex systems, then, organizations need to consider whether they have the “requisite variety” internally to meet the variety that is ever-growing in the external world.7D How does “requisite variety” translate into organizational language? It means taking typical organizational decisions such as hiring different people with more specialties in more places. By increasing the number of people in a particular function, widening the variety of positions, and encouraging the voicing of differing ideas, the organization invites some relative “chaos” into its orderly world, which, by its very nature, ultimately requires restructuring to accommodate it, to bring back more order.
Order itself, however, is not the final objective. The purpose is not to squeeze out as much chaos as possible, but rather to maintain just enough without losing order. The trick is the right balance between order and chaos.
Yaneer Bar-Yam,DD president of the New England Complex Systems Institute, describes the delicate point this way: “[The] balance between highly random and highly ordered motion is characteristic of the behavior of complex systems.” MacArthur Fellow Stuart Kauffman, one of the founders of the Santa Fe Institute, makes a similar point, pushing the balance towards maximum variety. Complex systems, Kauffman says, seek “an ordered regime, near the edge of chaos.”
4BCommunication and Decision-Making
Leaders make decisions then communicate them. Most executives depend on their organizational hierarchies to gather information, formulate options, offer recommendations, and make final decisions. Then they turn around and use the formal reporting lines as their primary conduits for distributing the official goals, strategies, policies, procedures, and other steers from the top. These directives eventually land in the laps of line managers who lead staff teams. This method of intelligence-gathering is changing, of course, with the rise of alternative input channels, largely the rapid dissemination of Web 2.0 technologies.
Regardless of how they gather and “promote” information to the executive level, organizations swing between decision-making, a comparatively complex process, and “decision-telling,” which depends on communication. Hierarchies need to accommodate both: a capacity for high-complexity decisions communicated along the shortest possible paths.
Which brings us back to organizational structure:
As they seek to optimize communication, organizations tend to centralize. The bigger the span, the fewer the levels, the shorter the communication paths.
To cope with complex decision-making, organizations tend to decentralize. The smaller the span, the greater the number of levels, the greater the capacity to make decisions.
In simplest terms: to communicate, flatten the structure—reduce the number of levels—and enlarge the size of the teams. To make better decisions, deepen the structure—increase the number of levels—and make the teams smaller. In the coming world, organizations will morph as needed to accommodate the pressing needs of the moment.
The dynamic of communicating and decision-making carves a hierarchical landscape that is high in some places, low in others. Many-tiered mountains of small decision-making teams optimized for complexity, like Obama’s foreign policy operation, are scattered through low-elevation plains of large teams transmitting strategies, standards, and procedures, like his much flatter communication operation. Extreme flatness, such as the McCain foreign policy operation, tend more toward chaos than order and likely was one source of his organizational problem.
Overall, organizations need to accommodate added decision-making capability while becoming even smarter about communicating. How can they do this?
The answer lies in how they mix the ingredients of organization size, reporting span, and number of levels. The twin dynamics of communication and decision-making provide the mixing agent.
5BSystems for Communicating
In social networks, “six degrees of separation” are said to connect any two random people in the world. In organizations, one degree separates each level: One link, one degree of separation.
Teams are ideal vehicles for communication. They are two-level organizations, a manager and his/her direct reports just one degree, one link, apart. Communication distance doesn’t get closer than one degree, whether in networks of family, friends, or coworkers. Every manager in the hierarchy has a one-degree team, a star-shaped cluster of closely related positions. A hierarchy is an interlocked set of one-degree management teams.
From a larger perspective, messages stream down the hierarchy of reporting links, a progressively articulated tree like any wide-area communication system. In cable transmission networks, for example, signals cascade from the “head end” (Level 1) through high-capacity trunk lines (Level 2) into lower-capacity branches (Level 3) and feeder lines (Level 4), finally “dropping” a thin wire to your home (Level 5). By analogy, the CEO is the head-end source of signal and content with managers in between “repeating” and “amplifying” the source transmissions who ultimately “drop” the messages at the “homes” of the staff.
Left to itself, the communication imperative will drive an organization to extreme centralization, a flat, star configuration of all one-degree links connected to one all-powerful manager – the McCain model. While this may be fine for an organization of five, it is problematic for a group of fifty or seventy-five, to say nothing of five hundred or five thousand and higher orders of magnitude.
To see how a structure optimizes for communication, imagine a “regular hierarchy” the same size as Eleum’s. By regular hierarchy, we mean that each manager has a fixed span, in this case six direct reports (approximately the same as Eleum’s average manager span of 5.8). A regular hierarchy of this sort is valuable for purposes of contrast to the real case because it represents most people’s idea of what a hierarchy looks like—in profile, a pyramid (See Myth #1 in NetAge Report #2). It is also easy to illustrate the interplay of the three basic metrics (size, span, and level) with a regular hierarchy.
A tree with a span of six requires six levels—five degrees of separation—to accommodate our target organization size of 5247, Eleum’s total number of employees in this dataset. A tree with a span of ten needs only five levels to get to the 5000+ number. A squatter, wider pyramid with plenty of room to spare, it contrasts with an organization that has a fixed span of three per team, which needs nine levels (a taller, skinnier pyramid) to reach the same organization size.
In Figure 2, we show a large group of engineers with similar specialization titles (e.g., Operations Engineer), an example of structure tuned to communication. Hyphenated titles (e.g., Operations Engineer-Facility A) often indicate a secondary reporting relationship elsewhere in the organization. Here, the large hub team aids the transmission of common standards, policies, and procedures. It also helps to circulate best practices for a group of experts. The leader acts more as the hub of a small community of practice than the maestro of an orchestra.
6BSystems for Making Decisions
Our primary everyday tool for tackling complex problems is analysis. “Breaking down” the problem divides something complicated into smaller, more digestible portions that may in turn be further divided. In organizational structure, this problem-solving capability shows up in the preference to differentiate and create more levels.
Bar-Yam points out how organizations respond as complex systems:
“As the collective behavioral complexity at the scale of an individual increases, the branching ratio of the control structure becomes smaller and smaller so that fewer individuals are directed by a single manager, and the number of layers of management increases. The formation of such branching structures allows an inherently more complex local behavior of the individuals and a larger complexity of the collective behavior as well.”
When complexity increases and an organization needs to make more decisions, it tends to decrease span and increase levels. Here, the hierarchy acts like a giant “decision-tree,” a method used by operations researchers to analyze complex choices. At the top (Level 1) is the final decision to make (e.g., allocation of resources among major projects), with branches (Level 2) to each of the major option areas. Operations researchers map out successive levels of branching and analysis within each option until they have calculated all alternatives and values. The more complex the choice, the more (dizzying) decision branches they need to map.
How should a problem-solving organization of twenty-five handle difficult decision-making tasks, the ones that require hard technical evaluations and lots of different specialties? What “depth” of organization will make it easiest for them? It is likely to be more effective with three or four levels of positions than it is as a two-level team. The communication cost of this arrangement, however, is that some positions are now two and three degrees of separation from the boss, increasing the principal complaint against hierarchy.
In Figure 3, we give an example from Eleum that illustrates a more articulated decision-making structure. A large group of engineers, sporting a variety of specialties and roles in their titles (such as architects, planners, process engineers, technology specialists), is organized in small teams within specialized departments. Here the leader is likely managing an interdependent set of specialized professionals who need to work together to produce the required output. The maestro analogy is appropriate for this type of organization.
Left to itself, the complexity-only imperative drives an organization to ultimate reductionism and extreme decentralization, a single chain of command dropping from position to position, in Eleum’s case, 5000-some levels. While single-person reporting chains of two or three degrees are surprisingly common, mile-deep vertical chains are hardly a great way to structure larger organizations.
But the trend is clear: organizational growth and larger size invariably require more levels, which inevitably increase the degree of separation among positions. More levels are on their way. And, once they’ve arrived, levels don’t go away easily.
7BBack to Reorganization
We opened our prior NetAge Report that analyzes the top levels of the US Government with this proposition:
Imagine that the standard mental model of organizations held by the vast majority of people, as we’re postulating, is really wrong, an epidemic distortion of reality. No wonder our large organizations don’t function very well, that bureaucracies rapidly grow out of control, and that frantic steering from the top seems to have so little effect. While people like to think the hierarchy is irrelevant, it is nevertheless unavoidable.
Without knowing any better, we seek an “ideal” flat regular hierarchy, seek to avoid “worse cases” of deep, detailed bureaucracies, and settle for something in the middle, a hierarchy-bureaucracy that is “the best we can do.”
Insights from study of hierarchies as networks can be brought to bear on significantly improving people’s ability to collaborate in large-scale structures. We must truly do “the best we can do” to develop the smart 21st-century networked organizations we so desperately need.
How might these results help answer the question we pose at the beginning: Is flatter better?
The risk an organization faces with a drive to eliminate levels is that it will severely damage its capacity to manage complexity. By shrinking levels in the push to optimize for communication, it may become too simple. Reducing levels—by chopping out managers and gathering people into ever-bigger teams—means more centralization. To take out multiple levels you will have to significantly change the size and distribution of spans.
BP may be correct in its overall diagnosis that its organization is too complicated and hard to manage, that its departmentalized specializations have grown beyond the capacity to communicate. Perhaps BP’s CEO, Tony Hayward, believes this will take his enterprise too far into the “hot zone” of decentralization, causing its leaders to attempt to manage too much variety requiring too many decisions. Adjusting the overall balance back towards the “cooler zones” of communication may be the right strategic design strategy.
But, potential suicide for a BP—or any large organization with properties similar to Eleum’s—is an ill-considered global mandate that overrides organizational design choices at the local level. Any imperative that forces sub-organizations to flatten and pushes teams to expand regardless of local circumstances is, in light of our analysis, foolish.
It’s smarter to give global guidance without inviolable imperatives. Ideally, create an accurate map of the whole so you can make local organizational decisions in a common context.
Organizations should not homogenize their zones of communication and decision-making capabilities. Understand, then enhance and improve them.
Hierarchies are not governed from or by the top. Rather, governance emerges from the network of teams—large and small—that result from myriad local design choices. These choices have been made over the years and are being made every day by leaders at all levels.
This is how change happens, little snippets of organizational evolution in action.