the art of scalability scalable web architecture processes and organizations for the modern enterprise phần 2 pps

These two descriptors, size and structure, will be covered in this chapter, providing insight into the various dimensions of each, large versus small team sizes and silo versus matrix st

100% consistent with the design For instance, if a design calls for a configurable

number (max number undefined) of similarly configured read databases, to which all

read transactions can be evenly distributed, and the software engineer implements a

system capable of handling up to five read databases, he or she has implemented a

system with a defined scale limit Here, defined scale limit is the limitation the

engi-neer put on how many databases can be implemented (five)

A software engineer should understand the portion of the system that she

sup-ports, maintains, or for which she creates code He should also understand the end

user and how the end user interacts with the software engineer’s portion of the

sys-tem The software engineer is a contributor to many of the scalability processes we

define later in Part II

Operator

The operator is responsible for handling the daily operations of the production

sys-tem, whether that system is a Web 2.0 system or a back office IT system She is

responsible for monitoring the system against specific service levels, monitoring for

out of bounds conditions, alerting individuals based on service level or boundary

condition failures, and tracking incidents to closure A form of operator, sometimes

called an incident manager, is responsible for managing major problems to closure

and issuing root cause and corrective action reports

Infrastructure Engineer

Infrastructure engineer is a generic term used to identify database administrators,

network engineers, and systems administration professionals The infrastructure

engineer is responsible for the selection, configuration, implementation, tuning, and

proper functioning of the devices or systems under his purview

The infrastructure engineer, more than any other role, is responsible for the

scal-ability of the systems that he supports As such, a database analyst is responsible for

identifying early when his database is going to fail based on capacity constraints and

to identify potential opportunities for scaling A systems analyst is expected to do the

same for her systems and storage and a network engineer for the portions of the

net-work that she supports

In addition to having a deep technical understanding of his specific discipline, a

skilled infrastructure engineer should understand the product he helps to support, be

conversant in the “sister” disciplines within the hardware and systems community (a

great systems administrator for instance should have a basic understanding of

net-works and a good understanding of how to troubleshoot basic database problems) in

order to aid in troubleshooting, a good knowledge of competing technologies to

those employed in his product or platform, and a good understanding of emerging

technologies within his field The infrastructure engineer should also understand the

cost of operating his system and the opportunities to reduce that cost overtime

Finally, the best infrastructure engineers are agnostic to the technologies they employ,

a point we will cover in Chapter 20, Designing for Any Technology

QA Engineer/Analyst

The QA engineer or analyst is responsible for testing the application and the systems

infrastructure to ensure that it meets the product specifications A portion of her time

should be dedicated to performance testing as it relates to scalability and as defined

in Chapter 17, Performance and Stress Testing

Capacity Planner

We’ve discussed the role and activity of the capacity planner in earlier sections of this

chapter Put simply, the capacity planner is responsible for matching the expected

demand (typically generated by the business unit) to the current system as

imple-mented to determine where additional changes need to be made in the system,

plat-form, or product The capacity planner is not responsible for defining what these

changes are; rather, she outlines where changes need to occur

In the case where a change needs to be made to a system that scales horizontally,

the capacity planner may have as part of her job description the responsibility to help

kick off the purchase order process to bring in new equipment More often than not,

the capacity planner is also a critical part of the process of budgeting for new systems

and new initiatives to meet the business forecasted demand

An Organizational Example

The new CEO of AllScale analyzes her team over the first 90 days The company has

had a number of scalability related incidents with its flagship HRM product and

Christine determines that the current CTO (in AllScale’s case, the CTO is the highest

technology management position in the company) simply isn’t capable of handling

the development of new functionality and the stabilization of the existing platform

Christine believes that one of the issues with the executive previously in charge of

technology was that he really had no business acumen and could not properly

explain the need for certain purchases or projects in business terms The former CTO

simply did not understand simple business concepts like returns on investment and

discounted cash flow Furthermore, he always expected the business folks to

under-stand the need for any of what business peers believed were his pet projects and

would simply say, “We either do this or we will die.” Although the technology team’s

budget was nearly 20% of the company’s $200 million in revenue, systems still failed

and the old CTO would blame unfunded projects for outages and then blame the

business people for not understanding technology

The CEO sits down with her new CTO, a person she picked from an array of

can-didates with graduate degrees in both business and electrical engineering or computer

science, and explains that while she will delegate the responsibility for technical

deci-sions to the CTO and empower him to make decideci-sions within his budget limitations,

she will not and cannot delegate the accountability for his results She explains that

she wants to create a culture of scalability in the company along the lines of the old

manufacturing mottos of “everyone is accountable for quality.” She will work to add

a nod toward scalability in the corporate vision and add a corporate belief

surround-ing the need to cost effectively scale to customer demands without quality of service

or availability (a.k.a downtime) problems

The new CTO, Johnny Fixer, asks for 30 days to review the organization, identify,

and put in motion some quick win projects and report back with a plan to make the

technology platform, organization, and processes highly scalable and highly

avail-able He promises to keep Christine informed and communicate the issues he finds

and concerns he has They agree to talk daily on the phone, exchange emails more

often, and meet personally at least once a week

Johnny quickly identifies overlaps in jobs in certain areas and responsibilities that

are completely missing from his team For instance, no one is responsible for

develop-ing a sdevelop-ingle cohesive capacity plan Furthermore, teams do not work together to

col-laborate on designs, architects are not engaged with the engineering teams and do not

understand the current status of customer grief with the product, and quality defects

are blamed on a QA team with no engineering ownership of bugs

Johnny works quickly to hire a capacity planner onto his team As it is May and

the company’s budgeting for the next year must be complete by October, he knows he

must get good data about current system performance relative to peak theoretical

capacity and start to get next year’s demand projections from the business to help the

CFO create his next fiscal year budget The newly hired capacity planner starts

work-ing with the engineerwork-ing team to install the appropriate monitorwork-ing systems to collect

system data in order to identify capacity bottle necks and she works with finance to

understand both the current budget and to help provide information to generate the

next year’s budget

Although the CTO is worried about all of his technology problems, he knows that

long term he is going to have to focus his teams on how they can work together and

create shareholder value He implements a tool for defining roles and responsibilities

called RASCI for Responsible, Accountable, Supportive, Consulted, and Informed

(this tool is defined further in the next section) and implements Joint Architecture

Design and the Architecture Review Board (defined in Chapters 13 and 14) to help

resolve the lack of cooperation between organizations

Johnny walks through the past 30 days of issues and identifies that the team is not

keeping track of outages, incidents, and their associated impact to the business He

makes his head of technical operations responsible for all outage tracking and

indi-cates that together they will review all issues daily and track them to closure He

fur-ther requires that all architects attend at least one of the daily operations meetings

per month to help get them closer to the customer and to better understand the pains

associated with the current system While meeting with his engineering managers,

Johnny indicates that all bugs will be considered engineering and QA failures rather

than just QA failure and that the company will begin tracking defects (or bugs) per

feature produced with a goal to reducing all failures

To help align his teams to the need for a more reliable and available site, Johnny

implements a site uptime or availability metric and a goal to achieve greater than

99.99% availability by month within the next four months With the CEO’s advice

and permission, and with the help of his architects, engineers, and infrastructure

engi-neers, he reprioritizes some projects to attack the site outage incidents that appear

(given the small amount of data) to have caused the most grief to the company

Johnny then implements a governance council for all engineering projects

consist-ing of the CEO, the CFO, and all of the business unit leaders The council is

respon-sible for prioritizing projects, including availability projects, and for additionally

measuring their returns against the promised success and business metrics upon

which they were based

After the first 30 days, Johnny covers his 30-, 60-, and 90-day forward plans with

the CEO and they jointly agree on a vision and set of goals for the engineering team

(see Chapter 4, Leadership 101) Christine then has an “all hands” meeting with the

entire company explaining that scalability and availability of the platform are of the

utmost priority and that it is “everyone’s job” to help ensure that the company and

its services scale to meet customer demands To help incent the company toward an

appropriate culture that includes the notion of being “highly scalable,” she insists

that all managers have as part of their bonus compensation a scalability related goal

that represents no less than 5% of their bonus She delegates the development of

those goals to her subordinates and asks to review them in the next 30 days

A Tool for Defining Responsibilities

Many of our clients use a simple tool to help them define role clarity for any given

initiative Often when we are brought in to help with scalability in a company, we

employ this tool to define who should do what, and to help eliminate wasted work

and ensure complete coverage of all scalability related needs Although technically a

process, as this is a chapter on roles and responsibility, we felt compelled to include

this tool here

The tool we most often use is called RASCI It is a responsibility assignment chart

and the acronym stands for Responsible, Accountable, Supportive, Consulted, and

Informed

• R stands for Responsible This is the person responsible for completing the

project or initiative

• A stands for Accountable This is the person to whom R is accountable and who

must approve the work before it is okay to complete The A is sometimes

referred to as the approver of any initiative.

• S stands for Supportive These people provide resources to complete the project

or initiative

• C stands for Consulted These people have data or information that can be

use-ful in completing the project

• I stands for Informed These people should be notified, but do not need to be

consulted or provide input to the project

RASCI can be used in a matrix, where each activity or initiative is spelled out along

the y or vertical axis of the matrix and the individual contributors or organizations

are spelled out on the x-axis of the matrix The intersection of the activity (y-axis)

and the organization (x-axis) contains one of the letters R, A, S, C, or I and may

include nothing if that individual or organization is not part of the initiative

Ideally, in any case, there will be a single R and a single A for any given initiative

This helps eliminate the issue we identified earlier in this chapter of having multiple

organizations or individuals feeling that they are responsible for any given initiative

By having a single person or organization responsible, you are abiding by the “one

back to pat and one throat to choke” rule A gentler way of saying this is that

distrib-uted ownership is ownership by no one

This is not to say that others should not be allowed to provide input to the project

or initiative The RASCI model clearly allows and enforces the use of consultants or

people within and outside your company who might add value to the initiative An A

should not sign off on an R’s approach until such time as the R has actually consulted

with all of the appropriate people to develop the right course of action And of course

if the company has the right culture, not only is the R going to want to seek those

people’s help, but the R is going to make them feel as if their input is valued and

value added to the decision making process

You can add as many Cs, Ss, and Is as you would like and as add value or are

needed to complete any given project That said, protect against going overboard

regarding who exactly you will inform Remember our discussion in the previous

chapter about people being bogged down with email and communication that does

not concern them It is common in young companies to allow everyone to feel that

they should be involved in every decision or informed of every decision This

infor-mation distribution mechanism simply does not scale and results in people reading

emails rather than doing what they should be doing to create shareholder value

A partially filled out example matrix is included in Table 2.1

Taking some of our discussion thus far regarding different roles, let’s see how

we’ve begun to fill out this RASCI matrix

We earlier indicated that the CEO absolutely must be responsible for the culture of

scalability, or the scale DNA of the company Although it is theoretically possible for

her to delegate this responsibility to someone else within the company from a

practi-cal perspective, and as you will see in the chapter on leadership, she must live and

walk the values associated with scaling the company and its platform As such, even

with delegation and as we are talking about how the company “acts” with respect to

scale, the CEO absolutely must “own” this Therefore, we have placed an R in the

CEO’s column next to the Scalability Culture initiative row The CEO is obviously

responsible to the board of directors and, as the creation of scale culture has to do with

overall culture creation, we have indicated that the board of directors is the A

Table 2.1 RASCI Matrix

CEO

Business

Board of Directors

Who are the Ss of the Scalability Culture initiative? Who should be informed and

who needs to be consulted? In developing your answer to this question, you are

allowed to have people who are Ss of any situation also be Cs in the development of

the solution It is implied that Cs and Ss will be informed as a result of their jobs, so

it is generally not necessary to include an I any place that you feel you need to

com-municate a decision and a result

We’ve also completely filled out the row for Technical Scalability Vision Here, as

we’ve previously indicated, the CTO is responsible for developing the vision for

scal-ability for the product/platform/system The CTO’s boss is very likely the CEO, so

she will be responsible for approving the decision or course Note that it is not

abso-lutely necessary that the R’s boss be the A in any given decision It is entirely possible

that the R will be performing actions on behalf of someone for whom he or she does

not work In this case, however, assuming that the CTO works for the CEO, there is

very little chance that the CTO would actually have someone other than the CEO

approve his or her scalability vision or scalability plan

Consultants to the scalability vision are the CTO’s peers—the people who rely on

the CTO for either the availability of the product or the back office systems that run

the company These people need to be consulted because the systems that the CTO

creates and runs are the lifeblood of the business units and the heart of the back

office systems that the CFO needs to do his or her job

We have indicated that the CTO’s organizations (Architecture group, Engineering

team, Operations team, Infrastructure team, and QA) are all supporters of the vision,

but one or more of them could also be consultants to the solution The less technical

the CTO, the more he will need to rely upon his teams to develop the vision for

scal-ability Here, we have assumed that the CTO has the greatest technical experience on

the team, which is obviously not always the case The CTO may also want to bring in

outside help in determining the scalability vision and/or plan This outside help may

be a retained advisory services firm or potentially the establishment of a technology

advisory and governance board that provides for the technology team the same

gov-ernance and oversight that a board of directors provides at a corporate level

Finally, we have indicated that the board of directors needs to be Informed of the

scalability vision This might be a footnote in a board meeting or a discussion around

what is possible with the current platform and how the company will need to invest

to meet the scalability objectives for the coming years

The remainder of the matrix has been partially filled out Important points with

respect to the matrix are that we have split up the tasks/initiatives to try to ensure

that there aren’t any overlaps in the R category For instance, the responsibility for

infrastructure tasks has been split from the responsibility for software development

or architecture and design tasks This allows for clear responsibility in line with our

“one back to pat and one throat to choke” philosophy In so doing, however, the

organization might tend to move toward designing in a silo or vacuum, which is

counter to what you would like to have long term Should you structure your

organi-zation in a similar fashion, it is very important that you implement processes that

require teams to design together to create the best possible solution Matrix

orga-nized teams do not suffer from some of the silo mentality that exists within teams

built in silos around functions or organizational responsibility, but they can still

ben-efit from RASCI You should still have a single responsible organization; but you

want to ensure that collaboration happens RASCI helps enforce that through the use

of the C attribute

Please spend time working through the rest of the matrix in Table 2.1 to get

com-fortable with the RASCI model It is a very effective tool in clearly defining roles and

responsibilities and can help eliminate duplicated work, unfortunate morale-deflating

fights, and missed work assignments

Conclusion

Providing role clarity is the responsibility of leaders and managers Individuals as

well as organizations need role clarity We provided some examples of how roles

might be clearly defined to help in the organization’s mission of attaining higher

availability We also argued that these are but one of many examples that might be

created regarding individuals and organizations and their roles The real answer for

you may vary significantly as the roles should be developed consistent with company

culture and need In attempting to create role clarity, attempt to stay away from

over-lapping responsibilities, as these can create wasted effort and value-destroying

con-flicts Also attempt to ensure that no areas are missing, as these will result in failures

We also introduced a tool called RASCI to help define roles and responsibilities

within the organization Feel free to use RASCI for your own organizational roles

and for roles within initiatives The use of RASCI can help eliminate duplicated work

and make your organization more effective, efficient, and scalable

Key Points

• Role clarity is critical for scale initiatives to be successful

• Overlapping responsibility creates wasted effort and value-destroying conflicts

• Areas missing responsibility create vacuums of activity and failed scale initiatives

• The CEO is the chief scalability officer of the company

• The CTO/CIO is the chief technical scale officer of the company

• Key scale related responsibilities for any organization include

qCreation of the scalability vision for the organization

qSetting measurable scale related goals

qStaffing the team with the appropriate skill sets necessary to meet the

scalabil-ity objectives

qDefining a scalable architecture

qImplementing that architecture in systems and code

qTesting the implementation against current and future user demand

qGathering data on current platform and product utilization to determine

immediate needs for scale

qDeveloping future demand projections and converting that demand projection

into meaningful system demand

qAnalyzing the demand projections against the system to determine where

changes are needed

qDefining future changes based on the analysis

qDeveloping processes to determine when and where systems will break and

prioritizing fixes for those issues

• RASCI is a tool that can help eliminate overlaps in responsibility and create

clear role definition RASCI is developed in a matrix in which

qR stands for the person Responsible for deciding what to do and running tive

qA is Accountable or the Approver of the initiative and the results

qS stands for Supportive, referring to anyone providing services to accomplish the

initiative

qC stands for those who should be Consulted before making a decision and

regarding the results of the initiative

qI stands for those who should be Informed of both the decision and the results

In the past two chapters, we have discussed how important it is to establish the right

roles within your team and to get the right people in those roles This hopefully

makes perfect sense and is in line with everything you believe: accomplishing great

things starts with finding great people and getting them in the right roles Now we

have come to the organizational structure, and you are probably wondering why this

has anything to do with the successful scaling of your application The answer lies in

what factors are affected by an organizational structure and in turn how important

those factors are to the scalability of your application

This chapter will highlight the factors that an organizational structure can

influ-ence and show how those are also key factors in an application’s or Web service’s

scalability There are two determinants of an organization: team size and team

struc-ture If you are given the size range of the teams and how the teams are related to

each other, you have a clear description of the organization These two descriptors,

size and structure, will be covered in this chapter, providing insight into the various

dimensions of each, large versus small team sizes and silo versus matrix structures

Organizational Influences That Affect Scalability

The most important factors that the organizational structure can affect are

communi-cation, efficiency, standards, quality, and ownership Let’s take each factor and

exam-ine how the organization can influence it as well as why that factor is also important

to scalability Thus, we can establish a causal relationship between the organization

and scalability

As we will see in Part II, Building Processes for Scale, which deals with processes,

communication is central to all processes Failed organizational communication is a

certain guarantee for failures in the application Not clearly communicating the

proper architectural design, the extent of the outage, the customer complaints, or the

changes being promoted to production can all be disastrous If a single team has fifty

people on it with no demarcation or hierarchy, the chance that everyone knows what

everyone else is working on is remote The possibility that an individual on this team

of fifty people knows who to ask what questions of or who to send what information

is unlikely These breakdowns in smooth communication, on most days, may cause

only minor disruptions, such as having to spam all fifty people to get a question

answered There will come a day when after being spammed for a year with

ques-tions that don’t apply to her, that engineer will miss a key request for information

that may prevent an outage or help to restore one quickly Was that the engineer’s

fault for not being a superstar or was it the organizational structure’s fault for

mak-ing it impossible to communicate clearly and effectively?

The efficiency, the ratio of output produced compared to the input, of both people

and entire teams can be increased when an organizational structure streamlines the

workflow or decreased when projects get mired down in unnecessary organizational

hierarchy In the Agile software development methodology, product owners are often

seated side-by-side engineers to ensure that product questions get answered

immedi-ately and not require long explanations via email If the engineer arrives at a point in

the development of a feature that requires clarification to continue, she has two

choices The engineer could guess which way to proceed or she could ask the product

owner and wait for the answer Until the product owner replies with an answer, that

engineer is likely stuck not being able to proceed and can either try to context switch

to something unrelated, which wastes a lot of time setting up environments and

restudying material, or she can go to the game room and waste a few hours playing

video games Having the product owner’s desk beside the engineer’s desk helps

main-tain a high efficiency by getting those questions answered quickly and preventing the

engineer from earning another high score on the video game The problem with

degrading efficiency in terms of scalability is what it does from an organizational

resource perspective As your resource pool dwindles, the tendency is to favor

short-term customer facing features over longer-short-term scalability projects That tendency

helps meet quarterly goals at the expense of long-term platform viability

The only standards that matter within an organization are those to which the

organization adheres An organization that does not foster the creation, distribution,

and acceptance of standards in coding, documentation, specifications, and

deploy-ment is sure to decrease efficiency, reduce quality, and increase the risk of significant

production issues Take an organization that is a complete matrix, where a very few

engineers, possibly only one, reside on each team along with product managers,

project managers, and business owners Without an extreme amount of diligence

around communicating standards, it would be very simple for that solo engineer to

stop following any guidelines that were previously established No other engineer or

manager is there to check that the solo engineer has not forgotten to submit his

docu-mentation, and the short-term gain in efficiency might seem a great tradeoff to him

The proper organization must help engineers understand and follow the established

guidelines, principles, and norms that the larger group has agreed to follow As for

the potential impact of this noncompliance on scalability, imagine an engineer

decid-ing that the architectural principle of all services must run on three or more

physi-cally different instances is not really important When it comes time to deploy the

service, this service can only run on a single server, thus increasing the likelihood of

an outage of that service as well as not being able to scale the service beyond the

capability of a single server

As described earlier, an organization that does not foster adherence to norms and

standards has the effect of lowering quality of the product being developed A brief

example of this would be an organization that has a solid unit test framework and

process in place but is structured either through size or team composition such that it

does not foster the acceptance and substantiation of this unit test framework The

solo engineer, discussed earlier, may find it too tempting to disregard the team’s

request to build unit tests for all features and forgo the exercise This is very likely to

lead to poorer quality code and thus an increase in major and minor defects In turn,

this can possibly lead to downtime for the application or cause other availability

related issues The resulting increase in bugs and production problems takes

engi-neering resources away from coding new features or scalability projects, such as

sharding the database As we have seen before when resources become scarce, the

tradeoff of postponing a short-term customer feature in favor of a long-term

scalabil-ity project becomes much more difficult

Longhorn

The Microsoft operating system code named Longhorn that publicly became known as Vista

serves as an example of the failure of standards and quality in an organization Ultimately,

Vista became a successful product launch that achieved the ranking of being the second most

widely adopted operating system, but not without significant pain for both Microsoft and their

customers The time period between the release of the previous desktop operating system XP

and Vista marked the longest in the company’s history between product launches

In a front-page article on September 23, 2005, in The Wall Street Journal, Jim Allchin,

Microsoft’s copresident, admitted to telling Bill Gates that “It’s not going to work,” referring to

Longhorn Jim Allchin continues in the article describing the development as “crashing to the

ground” due to haphazard methods by which features were introduced and integrated.1

1 Robert A Guth “Battling Google, Microsoft Changes How It Builds Software.” Wall

Street Journal, September 23, 2005, http://online.wsj.com.

One of the factors that helped revive the doomed product was enlisting the help of senior

executive Amitabh Srivastava who had a team of architects map out the operating system and

established a development process that enforced high levels of code quality across the

organi-zation Although this caused great criticism from some of the individual developers, it resulted

in the recovery of a failed product

The last major factor that the organization can affect that directly impacts the

application’s or service’s scalability is ownership If we take the opposite extreme of an

organization from what we were using in our previous examples, and assume an

orga-nization where each team has fifty engineers and no separation or hierarchy, we may

very well see many different engineers working on the same part of the code base When

lots of people work on the same code and there is no explicit or implicit hierarchy, no

one feels they own the code When this occurs, no one takes the extra steps to ensure

others are following standards, building the requested functionality, or maintaining

the high quality desired in the product Thus, we see the aforementioned problems

with scalability of the application that stem from issues such as less efficient use of

the engineering resources, more production issues, and poor communication

Thus, we see that the organization does affect some key factors that impact the

scalability of the application Now that we have a clear basis for caring about the

organization from a scalability perspective, it is time to understand the basic

determi-nants of all organization, size and structure

Team Size

Before we explore the factors that influence optimal team size, first we should discuss

why team size is important Consider a team of two people; the two know each

other’s quirks, they always know what each other is working on, and they never

for-get to communicate with each other Sounds perfect right? Well, consider that they

also may not have enough engineering effort to tackle big projects, like scalability

projects of splitting databases, in a timely manner; they do not have the flexibility to

transfer to another team because each one probably knows stuff that no one else

does; and they probably have their own coding standards that are not common

among other two-person teams Obviously, both small teams and large teams have

their pros and cons They key is to balance each to get the optimal result for your

organization

An important point is that we are looking for the optimal team size for your

orga-nization As this implies, there is not a single magic number that is best for all teams

There are many factors that should be considered when determining the optimal size

for your teams; and even among teams the sizes may vary If forced to give a direct

answer to how many team members should optimally be on a team, we would

pro-vide a range and hope that suffices for a specific enough answer Although there are

always exceptions even to this broad range of choices, our low boundary for team

size is six and our upper boundary is 15 What we mean by low boundary is that if

you have fewer than six engineers, there is probably no point in dividing them into

separate teams For the upper boundary, if there are more than 15 people on a single

team, the size starts to hinder the management’s ability to actively manage and

com-munication between team members starts to falter Having given this range, there are

always exceptions to these guidelines; more importantly, consider the following

fac-tors as aligned with your organization, people, and goals

The first factor to consider when determining team size is the experience level of

the managers The role of the engineering or operations manager can entail different

responsibilities in different companies Some companies require managers to meet

one-on-one with every team member for at least 30 minutes each week to provide

feedback, mentoring, and receive updates; others have no such requirement

Manage-rial responsibilities will be discussed later in this chapter as a factor by itself, but for

our purposes now we will assume that managers have a base level of responsibility,

which includes the three following items: ensuring engineers are assigned to projects,

either self directed or by edict of management; that administrative tasks, such as

resolving pay problems or passing along human resources information take place;

and that managers receive status updates on projects in order to pass them along to

upper management Given this base level of responsibility, a junior manager who has

just stepped from the engineering ranks into management may find that, even with a

small team of six engineers, the administrative and project management tasks

con-sume her entire day She may have time for little else because these tasks are new to

her and they require a significant amount of time and concentration compared with

her more senior counterparts who can handle all these tasks for a much larger team

and still have time for special projects on the side This is perfectly normal for

every-one New tasks typically take longer and require more intense concentration than

tasks that have been performed over and over again The level of experience is a key

factor to consider when determining the optimal size for a given team

In a similar vein, the tenure of the team itself is a factor to consider when

contem-plating team size A team that is very senior in tenure both at the company and in

engineering will require much less overhead from both management as well as each

other to perform their responsibilities Both durations are important, time at the

company and time in engineering, because they both influence different aspects of

overhead required The more time at a particular company will generally indicate

that less overhead is required for the administrative and human resource tasks that

inundate new people such as signing up for benefits, getting incorrect paychecks

straightened out, or attending all the mandatory briefings The more time practicing

engineering the less time and overhead that will be required to explain specifications,

designs, standards, frameworks, or technical problems Of course, every individual is

different and the seniority of the overall team must be considered If a team has a

well-balanced group of senior, midlevel, and junior engineers, they can probably exist in a

moderate-sized team; whereas a team of all senior engineers, say on an infrastructure

project, might be able to exist with twice as many individuals because they should

require much less communication about nondevelopment related items and be much

less distracted by more junior type questions, such as how to check out code from the

repository You should consider all of this when deciding on the optimal team size

because doing so will provide a good indicator of how large the team can effectively

be without causing disruption due to the overhead overwhelming the productivity

As we mentioned earlier, each company has different expectations of the tasks that

a manager should be responsible for completing We decided that a base level of

man-agerial responsibilities includes ensuring the following:

• Engineers are assigned to projects

• Administrative tasks take place

• Status updates are passed along

Obviously, there are many more managerial responsibilities that could be asked of

the managers, including one-on-one weekly meetings with engineers, coding of

fea-tures by managers themselves, reviewing specifications, project management,

review-ing designs, coordinatreview-ing or conductreview-ing code reviews, establishment and ensurreview-ing

adherence of standards, mentoring, praising, and performance reviews The more of

these tasks that are placed upon the individual managers the smaller the team that is

required in order to ensure the managers can accomplish all the assigned tasks For

example, if one-on-ones are required, and we feel they should be, an hour-long

meet-ing with each engineer weekly for a team of ten engineers will consume 25% of a

40-hour week The numbers can be tweaked—shorter meetings, longer work weeks—

but the point remains that just speaking to each engineer on a large team can

con-sume a very large portion of a manager’s time Speaking frequently with team

mem-bers is critical to be an effective manager and leader So obviously, the number and

effort of these tasks should be considered as a major contributing factor to the size

that the optimal teams should be in your organization An interesting and perhaps

enlightening exercise for upper management is to survey your front-line managers

and ask how much time they spend on each task for a week As we indicated with the

one-on-one meetings, it is surprisingly deceptive how easy it is to fill up a manager’s

week with tasks

The previous three factors, experience of management, tenure of team members,

and managerial responsibilities, are all constraints in that they limit the size of the

team based on necessity of maintaining a low enough overhead of communication

and disruptions to ensure projects actually get accomplished The next and last factor

that we will discuss is one that pushes to expand the team size This factor is the

needs or requirements of the business The business owners and product managers, in

general, want to build more and larger customer facing projects in order that they

continue to fend off competitors, grow the revenue stream, and increase the customer

base The two main problems with keeping team sizes small is first that large projects

require, depending on the product development life cycle methodology that is

employed, many more iterations or more time in development The net result is the same,

projects take longer to get delivered to the customer The second problem is that

increasing the number of engineers on staff requires increasing the number of support

personnel, including managers Engineering managers may take offense at being called

support personnel but in reality that is what management should be, something that

supports the teams in accomplishing their projects The larger the teams the fewer

managers are required Obviously for those familiar with the concepts outlined in the

Mythical Man Month: Essays on Software Engineering by Frederick P Brooks, Jr.,

there is a limit to the amount a project can be subdivided in order to expedite the

delivery Even with this consideration, it is still valid that the larger the team size the

faster projects can be delivered and the larger projects can be undertaken

The preceding discussion focused on what you should consider when planning

team structures, budgets, hiring plans, and product roadmaps The four factors of

management experience, team member tenure in the company and in the engineering

field, managerial duties, and the needs of the business must all be taken into

consider-ation Returning to our example at the concocted company of AllScale, Johnny Fixer,

the new CTO, had just finished briefing his 30-60-90 day plans Among these was to

analyze his organizational structure and the management team During one of his

ini-tial meetings, Mike Softe, the VP of engineering and a direct report of Johnny’s,

asked for some help determining the appropriate team size for several of his teams

Johnny took the opportunity to help teach some of his concepts on team size and

went to the whiteboard drawing a matrix with four factors across the top (manager

experience, team member tenure, managerial duties, and business needs) He asked

Mike to fill in the matrix with information about the three teams that he was

con-cerned about In Table 3.1, there are the three different teams that Mike is evaluating

Table 3.1 Team Size Analysis

Manager Experience

Team Member’s Tenure

Managerial Duties

Business Needs

For all the teams, Mike considered the managerial responsibilities to be medium

and consistent across all managers This may or may not be the case in your

organi-zation Often, a particularly senior manager may have special projects he is

responsi-ble for, such as chairing standards meetings, and junior managers are often required

to continue coding while they make the transition to management and hire their own

backfill Mike Softe’s Team 1 has a very experienced manager and a long tenured

team; and the business needs are high, implying either large projects are on the

road-map or they need them as soon as possible, possibly both In this case, Johnny

explains to Mike that Team 1 should be considered for a large team size, perhaps 12

to 15 engineers Team 2 has a very inexperienced manager and team members The

business requirements on Team 2 are moderate and therefore Johnny explains that

the team size should be kept relatively small, 6 to 8 members Team 3 has a very

experienced manager with a new team The business does not require large or

fre-quently delivered projects from this team at this time Johnny explains that Mike

should consider letting the team members on this team gain more experience at the

company or in the practice of engineering and assist by keeping the team relatively

small even though the manager could handle more reports In this case, Johnny states

with a smile, the manager might be a great candidate to take on an outside project

that would benefit the overall organization

Warning Signs

Now that we have discussed the factors that must be considered when determining or

evaluating each team’s size, we should focus on signs that the team size is incorrect

In the event that you do not have an annual or semiannual process for evaluating

team sizes, some indicators of how to tell if a team is too big or too small could be

helpful If a team is too large, the indicators that you should look for are poor

com-munication, lowered productivity, and poor morale Poor communication could take

many forms including engineers missing meetings, unresponsiveness to emails, missed

specification changes, or multiple people asking the same questions

Lowered productivity can be another sign of the team size being too large The

reason for this is that if the manager, architects, and senior engineers do not have

enough time to spend with the junior engineers, these newest team members are not

going to produce as many features as quickly Without someone to mentor, guide,

direct, and answer questions, the junior engineers will have to flounder longer than

they normally would The opposite is possibly the culprit as well Senior engineers

might be too busy answering questions for too many junior engineers to get their

own work done, thus lowering their productivity Some signs of lowered productivity

include missing release dates, lower function or story points (if measured), and

push-back on feature assignment Function points and story points are two different

meth-ods that attempt to standardize the measurement of a piece of functionality Function

points are from the user’s perspective and story points are from the engineer’s

per-spective Engineers by nature are typically over-optimistic in terms of what they think

they can accomplish; if they are pushing back on an amount of work that they have

done in the past, this might be a clear indicator that they feel at some level their

pro-ductivity slipping

Both of the preceding problems, poor communication and lowered productivity

due to lack of support, can lead to the third sign of a team being too large: poor

morale When a normally healthy and satisfied team starts demonstrating poor

morale, this is a clear indicator that something is wrong Although there may be

many causes for poor morale, the size of the team should not be overlooked Similar

to how you approach debugging, look for what changed last Did the size of the team

grow recently? Poor morale can be demonstrated in a variety of manners such as

showing up late for work, spending more time in the game room, arguing more in

meetings, and pushing back more than usual on executive level decisions The reason

for this is straightforward, as an engineer when you feel unsupported, not

communi-cated with, or that you are not able to succeed in your tasks, it weighs heavily on

you Most engineers love a challenge, even very tough ones that will take days just to

understand the nature of the problem At the point that an engineer knows he cannot

solve the problem, suddenly he falls into despair This is especially true of junior

engi-neers, so watch for these behaviors to occur first in the more junior team members

Now that we have covered some of the signs to look for when a team becomes too

large, we can shift our focus to the opposite extreme and look for signs when a team

is too small If the team is too small, the indicators to look for are disgruntled

busi-ness partners, micromanaging managers, and overworked team members If the team

is too small, one of the first signs might be the business partners such as product

managers or business development spending more time around the manager

com-plaining that they need more products delivered A team that is too small is just

unable to quickly deliver sizable features Another tactic that disgruntled business

leaders can take is instead of complaining directly to the engineer or technology

lead-ership, they focus their energy in a more positive manner by supporting budget

requests for more engineers to be hired

When a normally effective manager begins to micromanage team members, this is

a sign to look into the root cause of this behavior There might be lots of

explana-tions, including the possibility that her team size is too small and she’s keeping busy

by hovering over her team members, second guessing their decisions, and asking for

status updates about the request for a status update If this is the case, it is the perfect

opportunity to assign this manager some other tasks that will serve the organization,

help professionally develop her by expanding her focus, and give her team some relief

from her constant presence Some ideas on special projects include chairing a

stan-dards committee, leading an investigation of a new software tool for bug tracking, or

establishing a cross team mentoring program for engineers

The third sign to look for when a team is too small is overworked team members

Most teams are extremely motivated by the products they are working on and believe

in the mission of the company They want to succeed and they want to do everything

they can to help This includes accepting too much work and trying to accomplish it

in the expected timeframe When the team members start to leave later and later or

start to consistently work weekends, these are signs that you might want to

investi-gate if you have enough engineers working on this particular team This type of

over-working behavior is expected and even necessary for most startup companies but

working in this manner consistently month after month will eventually burn out the

team causing attrition, poor morale, and poor quality It is much better to take notice

of the hours and days spent working on tasks and determine a corrective action early

as opposed to waking up to the problem when your most senior engineer walks in

your office to resign

These are signs that we have seen in our teams when they were either too large or

too small Some of these symptoms of course can be caused by other things besides

team size but it is often the case that managers jump to conclusions with the most

cursory of investigations into the real cause and often do not even consider the

orga-nizational structure as a possible cause Often, the most frequently blamed is the

team leader and his inability to manage his team effectively or manage the customer’s

expectations effectively Before you place the blame on his shoulders, be sure that the

organization that you have placed them in supports them as much as possible by

assisting in his success Running a great junior leader out of management because

you placed him in a team that was too large for his current skill level would be

dread-ful Invest in the future of your leadership team by building the proper supporting

structure around each manager, allowing his own skills and experience to develop

Growing or Splitting Teams

For the teams that are too small, adding engineers, although not necessarily easy, is

straightforward The steps include requesting and receiving budgetary approval for

hiring, writing up the job description, reviewing resumes, conducting interviews,

making offers, and on boarding the new hires Although these simple steps may take

months to actually accomplish, they are generally easy to understand and fairly

sim-ple to imsim-plement The much more difficult task is to split a team when it has become

too large Splitting a team incorrectly can have dire consequences caused by

confu-sion of code ownership, even worse communication, and stress of working for a new

manager Every team and organization is different, so there is no perfect, standard,

one-size-fits-all way of splitting teams There are some factors to consider when

undergoing this organizational surgery in order to minimize the impact and quickly

get back to a productive and engaged existence among the team members

Some of the things that you should think about when considering splitting a team

include how to split the code base, who is going to be the new manager, what

involvement will individual team members have, and how does this change the

rela-tionship with the business partners The first item to concentrate on is based on the

code or the work As we will discuss in much more detail in Part III, Architecting

Scalable Solutions, this might be a great opportunity to split the team as well as the

code base into what we term failure domains These are domains that limit the

impact of failures by isolating services from one another

The code used to be owned and assigned to a single team needs to be split between

two or more teams In the case of an engineering team, this usually revolves around

the code The old team perhaps owned all the services around the administrative part

of the application, such as account creation, login, billing, and reporting Again,

there is no standard way of doing this, but a possible solution is to subdivide the

ser-vices into two or more groups: one handling account creation and login, the other

handling billing and reporting services As you get deeper into the code, you will

likely hit base classes that require assignment to one team or the other In these cases,

we like to assign general ownership to one team or even better to one engineer and

set up alerts through the source code repository that can alert each other if anything

is changed in that particular file or class, therefore everyone can be aware of changes

in their sections of the code

The next item to consider is who will be the new manager This is an opportunity

to hire someone new from the outside or promote someone internally into the

posi-tion There are pros and cons of each opposi-tion An external hire brings new ideas and

experiences; whereas an internal hire provides a manager who is familiar with all the

team members as well as the processes Because of the various pros and cons of each

option, this is a decision you do not want to make lightly and may want to ponder

for a long time Making a well thought out decision is absolutely the correct thing to

do, but taking too much time can cause just as many problems The stress of the

unknown can be dampening to spirits and cause unrest Make a timely decision; if

that involves bringing in external candidates, do so as openly and quickly as possible

Dragging out the selection process and wavering between an internal and external

candidate does nothing but cause trouble for the team

The last of the big three items to consider when splitting a team is how the

rela-tionship with the business will be affected If there is a one-to-one relarela-tionship

between engineering team, quality assurance, product management team, and

busi-ness team, this is something that will obviously change by splitting a team A

discus-sion with all the affected leaders should take place before a decidiscus-sion is reached on

splitting the team Perhaps all the counterpart teams will split simultaneously or

indi-viduals will be reassigned to interact more directly along team lines There are many

possibilities with the most important thing being an open discussion taking place

beyond the engineering and beyond the technology teams

We have covered the warning signs associated with teams that are too large and

too small We also covered the factors to consider when splitting teams One of the

major lessons that should be gleaned from this section is that the team size and

changes to it can have tremendous impacts on everything from morale to

productiv-ity Therefore, it is critical to keep in mind the team size as a major determining

fac-tor of how effective the organization is in relation to scalability of the application

Checklist

Optimal team size checklist:

1 Determine the experience level of your managers

2 Calculate each engineer’s tenure at the company

3 Look up or ask each engineer how long he or she has been in the industry

4 Estimate the total effort for managerial responsibilities

a Survey your managers for how much time they spend on tasks

b Make a list of the core managerial responsibilities that you expect managers to

accomplish

5 Look for signs of disgruntled business partners and managers who are bored to indicate

teams that are too small

6 Look for losses in productivity, poor communication, and degrading morale to indicate

teams that are too large

Splitting a team checklist:

1 Determine how to separate the code base

a Split by services

b Divide base classes and services as evenly as possible with only one owner

c Set up alerts in your repository to ensure everyone knows when items are being

modified

2 Determine who will be the new manager

a Consider internal versus external candidates

b Set an aggressive timeline for making the decision and stick to it

3 Analyze your team’s interactions with other teams or departments

a Discuss the planned split with other department heads

b Coordinate your team’s split with other teams to ensure a smoother transition

c Use joint announcements for all departments to help explain all the changes

simultaneously

Organizational Structure

The organizational structure refers to the actual layout or how teams relate to each

other within an organization This includes the separation of employees into

depart-ments, divisions, and teams as well as the management hierarchy that is used for

command and control of the forces There are as many different structures as there

are companies, but there are two basic structures from which everything else stems

These structures are functional and matrix By understanding the pros and cons of

each structure, you will be able to choose one or the other; or, perhaps a more likely

scenario, create a hybrid of the two structures that best meets the needs of your

com-pany In the ensuing paragraphs, we will cover the basic definition of each structure,

the benefits and drawbacks of each, and some ideas on when to use each one

Recog-nize that the most important lesson is how to choose parts of one versus the other to

create the organizational structure that is best for your scenario

Functional Organization

The functional organizational structure is the original structure upon which armies and

industries were based This structure, as seen in Figure 3.1, separates departments

Figure 3.1 Functional Organization Chart

Product MgmtCTO

Eng Team

Lead

Eng TeamLead

Eng 3

QA TeamLead

QA TeamLead

QA Eng 1 QA Eng 1

QA Eng 2 QA Eng 2

QA Eng 3

PM TeamLead

PM TeamLead

PM 3

and divisions by their primary purpose or function This was often called a silo

approach because each group of people was separated from other groups just as

grain or corn would be separated into silos based upon the type or grade of produce

In a technology organization, this structure would result in the creation of separate

departments to house engineering, quality assurance, operations, project

manage-ment, and so forth Along with this, there would exist the management hierarchy

within each department Each would have a department head, such as the VP of

engi-neering, and a structure into each department that was homogeneous in terms of

responsibilities Reporting into the VP of engineering would be other engineering

managers such as engineering directors and reporting into them would be engineering

senior managers and then engineering managers This hierarchy was consistent in

that engineering managers reported to other engineering managers and quality

assur-ance managers reported to other quality assurassur-ance managers

The benefits of the functional or silo organizational structure are numerous

Man-agers almost always were raised through the ranks; thus, even if they were not good

individual performers, they at least knew what was entailed in performing the job

Unless there had been major changes in the field over the years, there was very little

need to spend time explaining to bosses the more arcane or technical aspects of the

job because they were well versed in it Team members were also consistent in their

expertise, engineers worked alongside engineers Questions related to the technical

aspects of the job could be answered quickly by peers usually located in the next

cube This entire structure is built along specificity To use an exercise analogy, this

organizational structure is like a golfer practicing on the driving range The golfer

wants to get better and perform well at golf and therefore surrounds himself with

other golfers, perhaps even a golf instructor, and practices the game of golf, all very

specific to his goal Keep this analog in mind because we will use it to compare and

contrast the functional organizational structure with the matrix structure

Other benefits of the functional organizational structure, besides the homogeneity

and commonality of management and peers, include simplicity of responsibilities,

ease of task assignment, and the greater adherence to standards Because the

organi-zational structure is extremely clear, almost anyone, even the newest members, can

quickly grasp who is in charge of what team or phase of a project This simplicity

also allows for very easy assignment of tasks In a waterfall software development

methodology, the development phase is clearly the responsibility of the engineering

team in a functional organization Because all software engineers report up to a single

head of engineering and all quality assurance engineers report up to a single quality

assurance head, standards can be established, decreed, agreed upon, and enforced

fairly easily All of these are very popular reasons that the functional organization has

for so long been a standard in both the military and industries

The problems with a functional or silo organization include no single project

owner and poor cross-functional communication Projects almost never reside strictly

in the purview of a single functional team Most projects always require tasks to be

accomplished by multiple teams Take a simple feature request that must have a

spec-ification drafted by the product owner, a design and coding performed by the

engi-neers, testing performed by the quality assurance team, and deployment by the

operations engineers Responsibility for all aspects of the project does not reside with

any one person in the management hierarchy until you reach the head of technology,

which has responsibility over the product managers, engineering, quality assurance,

and operations staffs Obviously, this is a significant drawback having the CTO or

VP of technology the lowest person responsible for the overall success of the project

When problems arise in the projects, it is not uncommon for each functional owner

to place blame for delays or overspends on other departments

As simple as the functional organization is to understand, the communication can

be surprisingly difficult when attempting it across departments As an example, a

software engineer who wants to communicate to a quality assurance engineer about a

specific test that must be performed to check for the proper functionality may spend

the time tracking up and down the quality assurance management hierarchy looking

for the manager who is assigning the testing of this feature and request that she make

known who the work will be assigned in order that the information be passed along

More likely, the engineer will rely on established processes, which attempt to

facili-tate the passing along of such information through design and specification

docu-ments As you can imagine, writing a line in a 20-page specification about testing is

exceedingly difficult communication as compared to a one-on-one conversation

between the development engineer and the testing engineer

The benefits of the functional organization as just discussed include commonality

of managers and peers, simplicity of responsibility, and adherence to standards The

drawbacks include no single project owner and poor communications Given these

pros and cons, the scenarios in which you would want to consider a functional

orga-nizational structure are ones in which the advantages of specificity outweigh the

problems of overall coordination and ownership An example of such a scenario

would be a scalability project that involves sharding a database This is a highly

tech-nical project that requires some coordination among peer departments, but the

over-whelming majority of effort and tasks will be within the engineering discipline

Decisions about how to split the database, how the application will handle the

lookup, and all other similar decisions will fall to the engineering team The product

management team may be requested to provide information about specific customer

behavior or the cost to the business for changes to functionality, but it will not be as

involved as it would be for a new product line launch

Matrix Organization

In the 1970s, organizational behaviorists and managers began rethinking the

organi-zational structure As we discussed, although there are certain undeniable benefits to

the functional organization, there are also certain drawbacks Companies and even

militaries began experimenting with different organizational structures The second

primary organizational structure that evolved from this was the matrix structure The

principle concept in a matrix organization is that there are two dimensions to the

hierarchy As opposed to a functional organization where each team has a single

manager and thus each team member reports to a single boss, in the matrix there are

at least two dimensions of management structure, whereby each team member may

have two or more bosses Each of these two bosses may have different managerial

responsibilities—for instance, one (perhaps the team leader) handles administrative

tasks and reviews, whereas the other (perhaps the project manager) handles the

assignment of tasks and project status In Figure 3.2, the traditional functional

orga-nization is augmented with a project management team on the side

The right side of the organization in Figure 3.2 looks very similar to a functional

structure The big difference comes from the left side, where the project management

organization resides Notice that the Project Managers within the Project

Manage-ment Organization, PMO, are shaded with members of the other teams Project

Manager 1 is shaded light gray along with Engineer 1, Engineer 2, Quality Assurance

Engineer 1, Quality Assurance Engineer 2, Product Manager 1, and Product Manager 2

This light gray group of individuals comprises the project team that is working together

in a matrixed fashion The light gray team project manager might have responsibility

Figure 3.2 Matrix Organization Chart

Eng Team Lead

Eng 1 Eng 1

Eng 2 Eng 2

Eng 3

QA Team Lead

QA Team Lead

QA Eng 1 QA Eng 1

QA Eng 2 QA Eng 2

QA Eng 3

PM Team Lead

PM Team Lead

for the assignment of tasks and the timeline In larger and more complex matrix

organizations, many members of each team can belong to project teams

Continuing with the project team responsible for implementing the new billing

feature, we can start to realize the benefits of such a structure The two primary

problems with a functional organization are no project ownership and poor cross

team communication In the matrix organization, the project team fixes both of these

problems We now have a first level manager, Project Manager 1, who owns the

bill-ing project This project team will likely meet weekly or more and certainly have

fre-quent email dialogues, which again solves one of the problems facing the functional

organization: communication If the software engineer wants to communicate to the

QA engineer that a particular test needs to be included in the test harness, it is as

sim-ple as sending an email or mentioning it at the next team meeting, thus alleviating the

need to trudge through layers of management in search of the right person

We can pick up our golf analogy that we used in the discussion of the functional

organization You probably remember that we described a golfer who wants to get

better and perform well at golf To that end, he surrounds himself with other golfers,

perhaps even a golf instructor, and practices the game of golf, all very specific to his

goal This is analogous to the functional team where we want to perform a specific

function very well and so we surround ourselves with others like us and practice only

that skill What sports trainers have found out in the recent past is that specificity is

excellent at developing muscle memory and basic skills but to truly excel, athletes

must cross-train This is the concept of moving away from the golf course

periodi-cally and exercising other muscles such as through weight training or running This

cross-training is similar to the matrix organization in that it doesn’t replace the basic

training of golf or engineering but it enhances it by layering another discipline such

as running or project management For those astute individuals who have

cross-trained before, you might ask “can the cross-training actually hinder the athlete’s

per-formance?” In fact, if you are a golfer, you may have heard such talk around not

playing softball because it can cause havoc with your golf swing We will discuss this

concept in the context of the drawbacks of matrix organizations

If we have solved or at least dramatically improved the drawbacks of the

func-tional organizafunc-tional structure through the implementation of the matrix, surely

there is a cost for this improvement The truth is that while solving the problems of

project ownership and communication, we introduce other problems involving

multi-ple bosses and distraction from a person’s primary discipline Reporting to two or

more people—yes, matrix structures can get complex enough to require a person to

participate on multiple teams—invariably causes stressors because of differences in

direction given by each boss The engineer trapped between her engineering manager

telling her to code to a standard and her project manager insisting that she finish the

project on time is a setup that is asking for stress and someone not being pleased by

her performance Additionally, the project team requires overhead, as does any team

in the form of meetings and email communications This does not replace the team

meetings that the engineer must attend for her engineering manager and thus takes

more time away from her primary responsibility of coding

As you can see, while solving some problems, we have introduced new ones This

really should not be too shocking because that is typically what happens and rarely

are we able to solve a problem without consequences of another variety The next

question, given these pros and cons of the matrix organization, is “when would we

want to use such an organizational structure?” The appropriate times to use the

matrix structure are when the fate of the project is extremely important either

because of the timeline or other such factor In these cases, having a project team

sur-rounding the project, where there is a clear owner and the team structure facilitates

cross team communication, is the right answer to ensure delivery

Unfortunately you are likely to experience challenges across your organization

that are not always as straightforward as we have expressed in our simple examples

Real life, especially in a technology organization, is always more complex Here is

where the hybrid solutions become necessary Your entire engineering organization

does not need to be part of a matrix structure Instead, teams that are focused on very

cross team oriented projects may be in a matrix, but other teams working on

infra-structure projects might not be Alternatively, you could use the multidimensional

nature of the matrix without actually creating the project team An example of this

would be to collocate the project team together, in the same cube row, without

actu-ally implementing the matrix There are many other examples of how to create

hybrid functional-matrix organizations The key here is to use the organizational

structure to solve your problems that exist today There is no single right answer that

is right for all companies at all times

Conclusion

In this chapter, we highlighted the factors that an organizational structure can

influ-ence and showed how those are also key factors in an application or Web services

scalability Thus, we established a link between the organizational structure and

scal-ability to point out that, just like hiring the right people and getting them in the right

roles, building a supporting organizational structure around them is just as

impor-tant We discussed the two determinants of an organization: the team size and the

structure

In regards to the team size, we covered why size mattered—too small and you

can-not accomplish enough; too large and you lose productivity and impact morale We

further covered the four factors of management experience, team member tenure in

the company and in the engineering field, managerial duties, and the needs of the

business These all must be taken into consideration when determining the optimal

team size for your organization We also covered the warning signs to watch for to

determine if your teams were too large or too small For teams that were too large,

we stated that poor communication, lowered productivity, and poor morale were

symptoms For teams that were too small, we stated that disgruntled business

part-ners, micromanaging managers, and overworked team members were all symptoms

Lastly on the team size discussion, we covered what items to consider when growing

or splitting teams Growing teams was pretty straightforward but splitting up teams

into smaller teams entailed much more For splitting teams, we covered topics

includ-ing how to split the code base, who is goinclud-ing to be the new manager, what

involve-ment will individual team members have, and how does this change the relationship

with the business partners

The team structure discussion covered the two basic structures: functional and

matrix We described each, discussed the benefits, analyzed the drawbacks, and

rec-ommended scenarios to be used The functional structure was the original

organiza-tional structure and essentially divided employees up by their primary function, such

as engineering or quality assurance The benefits of a functional structure include the

homogeneity of management and peers, simplicity of responsibilities, ease of task

assignment, and the greater adherence to standards The drawbacks of the functional

structure were no single project owner and poor cross-functional communication

These problems were specifically targeted in the matrix organizational structure and

they were solved The matrix structure started out looking very similar to the

func-tional structure but a second dimension was added that included a new management

structure We provided examples of the matrix structure, which normally includes

project managers as the secondary dimension The strengths of the matrix

organiza-tion are solving the problems of project ownership and communicaorganiza-tion but the

draw-backs include multiple bosses and distraction from a person’s primary discipline We

concluded the organizational structure discussion with some thoughts on how hybrid

approaches are often the best because they are designed to fit the needs of your

organization

Key Points

• Organizational structure can either hinder or aid a team’s ability to produce and

support scalable applications

• Team size and structure are the two key attributes with regard to organizations

• Teams that are too small do not provide enough capacity to accomplish the

pri-orities of the business

• Teams that are too large can cause a loss of productivity and degrade morale

• Two basic organizational structures are functional and matrix

• Functional organizational structures provide benefits such as the commonality

of management and peers, simplicity of responsibilities, ease of task assignment,

and greater adherence to standards

• Matrix organizational structures provide benefits such as project ownership and

improved cross team communication

• Both team size and organizational structure must be determined by the needs

and capabilities of your organization