Lessons Learned in (Selling) Software Testing - Keith Klain Star East 2016

now it's my pleasure to introduce our

opening keynote speaker Keith claimed I

was going to talk about lessons learned

in selling software testing let me read

the official biography here for the last

20 years keith has built software

quality management and testing teams for

growth for global financial services and

IT consulting firms in the US the UK and

the asia-pacific region Keith designed

the software testing education program

with the bronx-based nonprofit purse

coalesce which has graduated over 150

students now from diverse backgrounds

into jobs in technology if you don't

know about that particular program you

really should look into it it's an

amazing thing that they're doing he was

executive vice president of the

Association for software testing and in

2013 the recipient of the software test

professionals luminary award it's my

pleasure to introduce keep claim that

sounded impressive better be good then a

how is everybody alright yeah excellent

it's nice to be back at star East

interestingly enough the last time I

spoke at star East was in 2012 doing a

keynote on the work I was doing at

Barclays and that my talk today is about

my transition from taking that model

that we introduced their kind of

enterprise-wide context-driven testing

approach to software testing and moving

from the buy side to the sell side but

first I wanted to give you all an update

I have joined a fantastic company based

out of New Jersey called tech mark so

that's where I'm at now they've been in

business for about 30 years there are

complete solutions and staffing business

I'm joining them to head up their

software quality management practice and

testing there are really fantastic

Rick and well-known particularly in

Northeast but they're also global

company they have an office in London

and I'm very honored to be joining the

management team and it's going to give

me an opportunity to take the work that

we've done in the past and expand it

they're very philanthropic organization

they've worked with preschoolers and

power and I'll be able to continue the

work that we've done there in the Bronx

and elsewhere across the country so I'm

very excited and happy can change my

title from consultants to whatever that

fancy title is I have now so but now on

to my disclaimers so I add things to the

beginning of my talks because I think

it's important for people to give

context and if people are the biggest

contributor to context it's important

that you know who I am and what my

context is so my context in software

testing is what I call enterprise

technology so this is large

organizations to believe financial

services or insurance that build

technology but their core business isn't

technology so that's like barclays ubs

Citigroup working as a consultant in

those areas and the reason why I think

that's important to give you that

context is that they typically don't

always use the best software methods

build methods they're usually large

diverse groups of people the enterprise

has a very unique set of challenges that

can really affect how you approach

software testing and particularly in the

software testing industry the enterprise

has been highly commoditized it's been

extensively offshored and generally has

a very low value proposition so that's

what my experience has been I haven't

worked that sexy technology companies

like Google or Facebook or anything that

I've worked at behemoths like Barclays

and Citigroup and they would I call it

sometimes is where a lot of good ideas

go to die so that's where my struggle

has been and that's that's where I'm

coming from so the next thing I liked

people is that I used to include this

slide in all my talks to poke fun at a

certain individual people who know me

well I will know who I'm talking about

but I think as well having moved from

the buy side to the sell side this

should include me as well so I'll read

that too if you don't know everyone know

who Daniel Kahneman is yeah if you

haven't read Thinking Fast and Slow you

should you should really read it it's a

great great book a lot of stuff in there

for software testers but the quote says

right overconfident professionals

sincerely believe they have expertise

act as experts and look like experts

you'll have to struggle to remind

yourself that they may be in the grip of

an illusion I think you know

particularly for the next couple days

that's probably a good framework to

judge the things that you're hearing

there's a lot of confirmation bias and a

lot of things that may be true to you or

your survivorship bias is kicking in but

it may not necessarily be true in the

classical sense of the word which leads

me to my next bit is that everything I'm

saying and probably what you've heard

from just about everybody else will

serve you very well you know good

testers have skepticism kind of

dyed-in-the-wool I think we shouldn't

accept what we see or hear at face value

I usually tell my kids believe none of

what you hear and half of what you see

but i think you know you should

definitely take what you're hearing and

putting it into your context and

treating well how that is relevant to

you so with that I also like to

delineate between what's your opinion

and what's your experience so I left a

great job running the global test center

at Barclays to try and expand the use

cases for context-driven testing and I

won't get into what that is or the

principles behind it you can do your own

research on that but it's a I think a

probably more true to the sense of the

word testing in the kind of scientific

method model approach to testing it's an

information-based and

text based approach to testing and so I

wanted to move beyond the transition

that we've made at barclays moving a

large enterprise towards that model and

see if we could do it in other

organizations so today I'm going to talk

about my opinions on some of the lessons

that I learned moving from the buy side

to the south side and I'm also going to

talk about there's four four main

lessons I learned then I'm also going to

talk about what I think can be done so

that you don't have to continually re

learn those lessons from my experience

so the kind of diagnosis is my opinion

and the the cure I think is is my

experience so after I left barclays and

started going out meeting with clients

and trying to talk them about

context-driven testing there is a

distinct feeling that came over me and

it's it was kind of hard to describe but

I think this is the best way to describe

it as anyone ever had that feeling

before raise your hand if you felt like

that before yeah thank you for raising

your hands thank you I'm not the only

one you know what I found was that

working in an enterprise environment had

kind of you know hardened my skull into

a uranium depleted helmet where a lot of

ideas I wasn't hearing and particularly

when you add in I was comping all these

people you know is probably in their

best interest as much as a skeptic as I

like to take myself to be when you're

paying people generally they're going to

try and agree with the person who's

giving you their their bonus check and I

know we like to talk about in tech a lot

about you know eating your own dog food

but you know it's it's really easy to

not really see how the dog food actually

tastes when you're paying everyone and

you know in confirmation bias is also a

really powerful drug if you've got a

touch of imposter syndrome as well where

you know pasta syndrome is you know high

achieving people feel like they

don't know what they're doing and I

think particularly in software testing

it's really pervasive because it's a

very weird business that we're in and

there's a lot of mixed messages that

come from vendors and consultants and a

lot of times you can feel like you don't

know what you're doing in this business

and particularly for people who come at

it from all sorts of different

backgrounds so that really wasn't a

lesson that was more of a realization so

the first lesson that I really learned

is that now despite me seeing this in

the enterprise or at Barclays other

companies as well are really horrible at

defining their objectives for why

they're testing and I you know this is

one of those things that I always felt

to be true you guys watch Bill Maher you

know he says I can't prove this but I

know it's true well getting out and

talking to clients and and meeting other

large financial institutions and doing

reviews of their testing approach people

still are not aligning their test

approach to their business you know

we've done lots of reviews of automation

and the first question I used to ask

people want to reviewed test automation

frameworks at the the various companies

I've worked at is well what would happen

if we stopped running this and you

generally get the RCA dog ooh you know I

don't know I'm so okay well next

question is what does it do and you

still get a lot of oh not really sure

you know and that's that's there is so

much sunken cost bias and a lot of the

test automation that's done and

unfortunately I'm probably getting hate

tweeted right now as being a Luddite you

know or are actually hating test

automation but it's not true I think

there's a lot of value to be achieved

from it but we just kept coming across

organization after organization that

we're just running and building these

things I talked to a financial services

company in Connecticut and yeah I accept

on with the engineers and said how do

you go about defining which tests you're

going to run what which tests are going

to automate and I appreciated the candor

but the

the guy said well we just see if we can

not should we or is there any value to

it people are still making these kind of

bad decisions and particularly the thing

that I'm coming across as well recently

as large enterprises that are trying to

transition to agile or they've made a

lot of investments in co ease and in

centralized testing and then suddenly

they're bolting on a whole mobile or

digital division to that that has a

completely different paradigm in terms

of how software is built and deployed

and they can't reconcile the two and so

you find this you know kind of non

congruence between the two sides of the

organization the other side of this as

well and I I did this at the eurostar

when Michael was was chairing I went

around the expo and wrote down about

well actually had to stop writing

because I ran out of paper of all the

unverifiable claims made by our industry

and yeah we are not helping ourselves in

terms of vendors I mean there's I've got

a little pet tweet project right now

with this company that keeps tweeting

out bug free software you know and

that's just that we're not helping that

night and I use this term all the time

that the amount of algorithmic defect

predict inators that gets old to CIOs

and CEOs greatly undermines our business

you know I give this slide and some of

the talks I do about how to increase

your value as a software tester and the

slides been in there for it was based on

a report that was done in Europe

probably five six years ago about they

pulled a thousand CEOs about how they

felt about their technology and one of

the questions was how do you feel about

your testing team and it was all the

classics too slow cost too much money

more automation if we could just fire

all these damn people would be so much

easier you know and and honestly that

case really hasn't changed that much so

as an industry we haven't really done a

lot to improve our value proposition the

next lesson that I learned was that what

that generally results in is

highly descriptive slides is how people

giving the enterprise treat testing

which is basically what i described

poking the testing robot with the metric

stick and it's actually probably

probably worse than that is as you know

i'll paraphrase james bakri says they

actually don't want a robot what they

want is a human-based robot skin for a

human to get into to then act like a

robot and I butchered that James I'm

sorry but like I said I think that you

rail against the dying of the light but

I just saw a report that came out the

other day from a company I used to work

at talking about their co e and how they

were going to you could have time-warped

it back 20 years how that you know why

aren't we having less people why isn't

automation saving us time why all these

investment why aren't our vendors

driving down our cost base because we've

offshored all this stuff and it

literally was like a white paper for all

the things that bad approaches the

software testing in this model are the

direct result from you know and and it's

just literally looking at why aren't

things faster cheaper we're measuring

the hell out of this counting all our

test cases why aren't we getting better

and this is a bit this is still very

valid today which if you see how these

lead into each other what creates in a

lot of enterprise testing organizations

is a really simple problem you know we

create a third culture you know and this

is particularly bad in co ease or

organizations that have highly offshored

forget about outsourcing but I've

offshored there they're testing teams

you know and and so in response to that

what I've seen is that and this has been

around for a while is that the testing

community you know creates our own

language and instead of learning the

language of our business or our projects

we create a jargon filled

old that we can communicate with each

other in fact we don't really agree what

we call things either and it creates

this kind of sub culture where people

look at us like aliens and you walk up

the people and start talking about

defect leakage and other things and you

know we can't even agree what it is and

then you just people tune out

immediately and I think there's too much

focus on trying to TT to teach people

about about what software testing is as

opposed to trying to increase our value

proposition and articulate what that is

you know I always set a good barometer

for me and I gave a talk about this last

year's how to talk to a CIO about

software testing and my answer to that

is don't I don't want to talk to my CIO

about software testing that's that's my

domain that's that's my responsibility i

want to talk to the cio about the

business about how i'm adding value how

i'm helping protect the brand and and

mitigating risk so i think this is you

know we tend to insulate ourselves and

create this third culture and so which

in in my context which is particularly

the skilled testing or context-driven

testing community this creates a

reaction as well that you know i believe

has tended us to be what i call smarty

pants syndrome and you know where we

feel like we're undervalued people don't

understand what we do we've created this

language that we don't think people

understand so we tend to kind of be

jerks about it and you know the one

other thing I found to trying to recruit

context-driven testing test leads people

who identify with the community and who

can manage a project or manage a group

of testers is that they're incredibly

hard to find and I think there's two

reasons for that one and I'll again

paraphrase James Bach as well is what he

calls the tiger cub problem where you

know you teach people how to test better

use their brains you unleash the power

of creativity and all these other things

and they grow these claws you know but

they don't know how to kill the bunny

right so they're they're walking around

slashing at everything and they're

arguing with people when you teach

people how to argue you also need to

teach them how to pick their battles and

I think what I found through hiring in

and and really kinda letting some some

test leads go is that there's a lot of

work that needs to be done in our

community on giving people the right we

do I guess in the workforce development

world you'd call it train and pray where

you give people skills and hope they

know how to use them and the other thing

about this I think the community is

relatively young there's a lot of

immaturity so if you've got people who

want to be skilled testers who want to

use their brains it's going to attract a

certain type of person that probably is

going to be more aggressive I think

that's but that was a particularly hard

problem and turn through a lot of lot of

test leads the other thing about this is

that people get allergic two words which

and I think maybe I've been around too

long to care too much about it but you

can give you know people if you say if I

said test case you know probably half

the people in this room spine just

stiffened you know or the other halfs

like yeah you know X lutely test cases

you know our test scripts is another one

you know or I've seen and and you know a

good good friend of mine sitting up here

will can relay the story at some point

about this passive aggressive and

sometimes just aggressive aggressive

word correction and I've seen people

have conversations where you know

they're the one person who's talking

about software testing and the other

person saying QA and they'll say well

did you QA that I was like well yes I

tested it well what is it going to come

out of QA well when we finish testing it

and you know I want to be specific in

language and I think there's a lot of

value in that but i also don't just want

to piss everybody off and I think

sometimes that's what that does so I

think as a community and I'm going to be

very careful how I say this because the

last time I gave this talk I got tweeted

very in out of context and got flamed

for a

after it but what I'm going to say now

is that the as a community I think we

need to do a better job shepherding

people into this community around our

values and principles about how we treat

people not just how we use language and

conduct software testing but also how we

treat people the values and principles

that come with that not just focus on

the ceremonies so those were the four

main lessons that I learned about

objectives smarty pants syndrome third

culture and and this is where I felt

like I was and I and this slide I've

used a lot in an introductory to

software testing class that I give to

the purse coalesce grads and does ever

know what this is so that's us Edmund

Hillary sir edmund hillary and tenzing

norgay the very first time they summited

everest and you know they didn't have

the benefit of you know ladders roped up

over the ice falls and things like that

they were up there with rope and and

each other essentially and when you the

first time they summited at about 28,000

feet is what's now called the Hillary

step the first time they encountered it

was just basically called a 40-foot

sheer rock face and so they get to

28,000 feet and they got long I can just

imagine the what the feeling they had

when they got there so they figure out

how to scale this thing get to the top

of that and I think great we're at the

summit oh my god no we're not there is

another thousand feet of the worst slow

flat drudgery up to the actual summit

and so this is a metaphor for how I

describe a career in software testing

two new testers

and yet we've graduated 150 people in

the news are missing gaps the point I'm

trying to make is that software testing

is I you could make an argument a

relatively new career there's a lot of

learning and and discover being done you

know around the language and how we do

this and your career is not going to be

just enlightenment throughout the entire

thing there's going to be periods of we

get to a you know a plateau I I

understand something I get over that

step in and wait I think I figured it

out all wait a minute that's not called

that anymore i knew something new i find

out and i used this to to help software

testers and new software testers guide

them through that process that it's not

all figured out that there is a lot to

learn yet in this business and and again

with my kind of you know smarty pants

had on thought it was mainly about other

people and i really learned it was

primarily about me so that's where i am

right now with everyone else so these

are so now i want to get into what i

view as my experience in some things

i've learned over the last couple years

that are going to i believe going to

help me specifically because it's all

about me prevent me from relearning some

of those those those lessons again and

the first one is and i've talked a lot

about this is if you to combat poor

objectives if you cannot draw a line

directly between your business and when

i say your business i mean like what

your business actually does and how

you're supporting it directly to what

you're doing every day in my experience

I think you're doing it wrong right and

this is how I view that that hierarchy

the business first then the technology

that supports it the programs right how

you test and then your own personal

objectives one I think that's just a

really good career path to be if you

can't constantly refocus what you're

doing towards how you're helping the


I I think you can go down rabbit holes

you can get that's why I think a lot of

automation projects who start out you

know on the path to hell with the best

intentions go go go off course rather

quickly is that we're not refocusing

what we're doing on how we're directly

helping the business you know I used to

do this at Barclays all the time and

I've you know use it as a metaphor now

but we would call it the critical defect

test so if our CIO walked into the room

and asked you you know stand up and tell

me the last critical defect you found in

test and what its business impact would

have been if it would have got into

production right tell me right now stand

up so you don't have to stand up right

but I would do that in what in our in

our testing groups and if you couldn't

do that I think you're not focusing what

you're doing on on the business you know

I how do you get this information and

I've given this advice of all I need to

actually track down the gentleman that

gave it to me years ago was that you

have particularly in the enterprise

which are typically your publicly traded

companies go pull your company's 10k

right and look up what that will have

all the risks their business objectives

their competitors information all that

stuff it's just like a target-rich

environment for test ideas and they're

all going to be focused on the actual

business you know and and I think as

well people say testing is a service but

it's not a servant right so we should be

objectively objectivity as a tester as

part of your responsibility and I think

that will help you refocus on providing

objective information while not solely

focus on trying to get a product or

project out now that may sound like

those is counterintuitive but I i

personally don't believe your sole

objective as a tester is to help

products get out your ear to provide

information and a tool I think that you

can use to to help with with learning

the business is what's a fantastic book

that you know Michael bolt

put me on to which is rethinking

expertise which is about interaction

elects purty switch and again at the

risk of insulting I'll just read its

expertise in the language of a

specialism in the absence of expertise

in its practice and if you think of

expertise on a continuum where on one

end you've got kind of beer mat cliff

clavin from Cheers you know knowledge to

the other side where you're really

contributing to the science you are at

that white-hot core of actually moving

the science forward interaction of

expertise is being able to talk to

people in across that spectrum while in

their language not in yours and and I

have to give up another credit to Martin

hen yay who put me onto a book called

training zones and interaction elects

purty by michael gorman with how to put

some of that stuff into practice those

are two very good books they're not huge

that you should read but I truly believe

you should learn the language of your

business and speak that and your

projects as opposed to contribute

through forcing our jargon on people so

if you want to you know I guess what's

the saying the the beatings will

continue until morale improves if you

want to you know take a few take a few

less wax from the metric stick I I

believe that test management is eighty

percent hands-on and I think one of the

Bane's of our industry is what I refer

to as operational test management which

is managers of managers or as i like to

call them at ubs the coloring in boys

these are just people who produce

metrics reports who are just kind of

looking at spreadsheets all the time you

know and so I at to give you a frame for

this at Barclays I had 1,700 testers

working for me across the whole of the

investment bank and wealth management I

manage that entire group with five

managers everybody below those five

managers tested eighty percent of the

time and so what I what I would what I

put into the definition of

test management for me is also things

like you know obviously hands-on testing

but I believe that mentoring is part of

hands-on testing sitting next to people

pair testing coaching watch me do it now

you do it all these things that that you

can get involved in from a hands-on

perspective are included in that

paradigm but if you are spending more

than twenty percent of your time on

reporting or progress or any of that

stuff I think it's that's a big red flag

for me to rethink how you're actually

managing the software testing so the

third culture and this is going to be my

advice just a tick list of things that

again they're heuristics for me for a

high-functioning software testing

program in the enterprise and the 30 70

rule for me is I would like to see

thirty percent perm staff I'm sorry

thirty percent external staff and

seventy percent perm staff which is

generally on its head from what most

enterprise groups run I've been in this

business for nearly or over 20 years now

I have been sold never bought but sold

countless CEO ease I don't know what

they are center of excellence a--'s I

don't know how they operate so I i think

the paradigm in itself creates barriers

that don't need to be there so my advice

would be not to do a co ii you know

constantly refocusing your business

perspective or reorganizing to make sure

that reliable is the time it takes us to

start testing because I and in all its

guises whether that's you know testing

our requirements testing our use cases

you know or actually physically testing

things when did we start asking critical

questions about this program you can

kind of mark how that because that will

give you an idea of how much test prep

and work is required to start getting

information back to the program but

that's that's usually it I've

ever seen and and I I put the

responsibility at the feet of the

testing business the testing industry

for why testers are continually asked

for all these crazy metrics is because

the CIOs and CEOs and and program

managers are sold they're constantly the

echo chamber of the software testing

industry is constantly ringing with the

siren call to count things there's your

insight James all right so

intermediaries I wouldn't use those and

that what i mean by that is if

particularly if you've got an offshore

team or a centralized testing group

where you know joe bloggs you know

program manager talks to jane blogs test

manager and in a go talk to their

respective teams i I firmly don't

believe that because you should manage

things you know together as a team and

as well if you do have a central test

group keep all the middle management

away from it because they will generally

just try and find things to do and

create a lot of havoc for what you

really want is a tightly integrated test

approach with your program so combating

smarty pants syndrome you know I like

this quote from from the good doctor

that we're never right we can only be

sure that we're wrong you know and I

think particularly around language or if

you're trying to help people and this is

one of the things that I've continually

tried to coach test leads on is if you

want somebody to do something the

Eisenhower quote because you want them

to do it you want somebody to do

something you want because you want them

to do it why do we instantly start

beating them over the head with the

language that they use and this is about

test cases and test scripts and QA and

again I'm not a fan of in precise

language but I think it's important to

think of yourself particularly if you're

trying to improve your lot in life as a

tester think of yourself as a GP not as

a GP but as a TR surgeon right so we're

in you know the meatball surgery from

mash and probably dating myself with


these analogies you know because again

correcting language just just upsets

everyone and you're already dealing with

somebody who's sick let's not beat them

up over the fact that they're sick right

they already know that or maybe they

don't and again I saw this great quote

from Benson mode the other day on

Twitter he said it was a big difference

between impostor syndrome and just being

an imposter alright so how do you go

about this and I try and include these

in my life as well as how I approach my

my job which again I think particular

over the last couple years has really

made a difference in how I help people

get better at software testing the first

one is you know humility and does ever

know who Neil deGrasse Tyson is yeah

bunch of testers you probably do you

know he talks a lot about the cosmic

perspective and being in awe of the

universe and not making not that not

making you angry or feeling

insignificant but feeling empowered and

humble and all that in the face of all

that and I think there's a lot of good

analogies between that and software

testing the you know impossibility of

complete testing just the pure maths

around some of this stuff you know

asking you know peering into the abyss

of infinite possibility and being asked

to figure out whether a product is safe

to ship or not you know that's quite a

daunting task so being humble in that

and not acting like you know all the

answers already i think is a great

position to start talking to people out

testing you know being self-aware how

you're contributing to the problem and

you know i think this is you know the

first question I've always asked

whenever I've gotten involved in a

management dispute or or anything with

my teams is you know what are you doing

to contribute to this problem and being

self-aware as an industry of what we're

doing if we feel like testing isn't

valued and this is like pervasive in

what's been written about software

testing what why is that you know are we

doing that and I believe firmly that the

answer to those questions lies and how

we're acting I talked about you know

being a

nervous but not a servant you know

inclusion and for as far as testing is

concerned to me is about letting people

understand how hard it is you know what

we're trying to do and that we're trying

to you know protect the company and risk

but it's actually very difficult that's

why I think pair testing is a great

great process but also from how we

develop test ideas and that's why I

really like you know it's it's not a

practice based approach but one of the

things that I learned about greatly

through the the context driven testing

community is using alternative methods

for test strategies you know at barclays

we coined the term visual test visual

test models where we did a lot of

modeling either on whiteboards and use

that and you can get people in a room

and collectively identify risk as

opposed to trying to document it in a

giant you know 20 page test strategy

nobody's going to read and I think the

last part there is about being

context-aware and this is again a kind

of mild criticism I think of the the

context-driven testing community is that

for all our talk about that I see a lot

of people who don't include the context

there in in terms of the people and

company as opposed to just focusing on

practices and I think that is probably

one of the biggest contributors to

helping people get better at software

testing or at least you know from the

most lowest level making your life

easier understanding who the people are

you work with and how you know they're

contributing to the the culture around

testing and what you can do to improve

that so and lastly and this is going to

be a bit of you know eating my own dog

food here you know manage your own

expectations you know I I recently wrote

an article for fantastic testing

magazine called testing trapeze and I

called it the you know dr. strange

career or how i learned to stop worrying

and love the software testing industry

and it was after going to the QA

symphony user conference and listening

to Scott Birkin who

a fantastic author and speaker and lots

of stuff that he talks about the

innovation that you should you should

read and listen to him talk is fantastic

but they he gave the keynote there and

it was a little bit of an epiphany for

me that you know and it's going to sound

funny but you know why am I always feel

like I just went got this business you

know it's I we don't make it any

improved so hard and and he talked about

you know Magellan and you know what's

Magellan known for circumnavigating the

globe right well he actually didn't you

know he died before he did and you know

he says like well why you know but he's

generally given credit for and the

thought occurred to me like well you

know of course because he was trying to

circumnavigate the globe like the you

know 16th century dummy you know it was

hard what he was doing and a lot of what

we do in software testing is is is very

hard and you know I think personally

that's that's what makes it great so

with that thank you for listening to me

and that's me done