Loading presentation...

Present Remotely

Send the link below via email or IM

Copy

Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.

DeleteCancel

Asking Better Questions Consulting

Training for analytical planning and consulting
by

Jeffrey Ryan

on 11 January 2017

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of Asking Better Questions Consulting

Analytic Planning and Consulting
Clarifying the Charge
What is a Charge?
The customer's statement of the requirement: the gap in knowledge between what the customer currently knows and what they want to know.

Exploring the Customer Need
Consider contextual concerns:

Think through how the customer's Need reflects global issues, events and trends. These provide context for their individual concerns, their organizational concerns, and the company's national concerns.
Assessing the Fit Between Charge and Need
How can you check that the Charge properly fits the customer's Need?
Place yourself in the customers shoes and imagine that the Need is your own
Write out the Charge you would have created to address the Need
This is your educated guess at the "implicit charge", now compare this with actual Charge and note the differences
Decomposing the Questions
What is my Charge asking me to do?
You will have to make a number of decisions about the appropriate scope and measurement for your analysis, and about the underlying hypotheses you will be trying to test or evaluate.
Clarify the Charge first by asking:
Then consider these three aspects:
Scope
What should be included, and what should be left out?
People:
Individual, Group
Places:
City/Country, Region
Things:
Objects, Causes, Organizations
Actions/Events:
Incidents, Changes
Time:
Calendar, Lifespan
Devising Analytic Strategy
The process of
Devising Analytic Strategy
will be quite specific to your particular Charge, and does not lend itself to generic prescriptive rules. The concept of Charge type provides a useful point of departure for this process, once you have determined which type you have, it is easier to think about how you should approach the data you will use for your analysis. With some overlap, different Charge types correspond to different analytic functions, or means of extractive evidence from data.
Tradecraft Tips: Conversations to Clarify the Charge
Identifying the "Critical Few" Questions
What is the most ambiguous about the Charge?
What are the biggest obstacles to a clear understanding of the Charge?
Delivery
Use focused Closing questions with an emphasis on specificity, e.g.,
" What is your definition of 'X'?" "Do you want me to Y?" "Is this Z?"
Use phrasing that will encourage precise answers, e.g.,
"Did you mean A or B," rather than " What did you mean by X?"
Be Concise and efficient, using your time carefully.
Live Calibration
Set the agenda quickly and clearly, e.g.,
"If you have a few minutes, there are two our three things I'd like to talk with you about. The first is X, the second is how it relates to Y, and the third is about where and whether Z fits in as well. I know that you'll need to run in 15 minutes. Does this sound like something we can cover with the available time?"
Steer the conversation but remember to avoid forcing every response you receive into any mental models that gave rise to your initial questions. Some example of was to steer the conversation effectively include:
Confirm comprehension
:
"It sounds like we've covered X. Are we ready to go to Y?"
Narrow scope:

"You're talking about X and Y. It would help if I knew a little more about X." / "What, specifically. about X concerns you?" / "Who in particular is asking?"
Widen scope
:
"It would help if I knew more about the background to X and Y." / "Tell me more about X." / "What else does X do?"
Close a topic
:
"It seems that we've covered everything regarding X for now." / "I think I now understand what I need to do about Y."
Test for thoroughness:

"Is there anything else I should know about X?"
Conduct a reality check
:
"I'm not sure this exact information exists. Is there an alternative?"
Be careful not to fall into rapid-fire interrogation.
Restate you understanding of the Charge an the end for clarity.

Topics(s)
What is the overall subject that the customer is interested in?
Definitions
Do any of the customer's terms need clarification?
How do contextual concerns shape the customer's Need
Thinking through these different but interrelated layers can help you better understand the customer Need from several different angles.

It is often tempting to fixate on one level, an awareness of the broader context can help to see aspects of a customer's Need that might go unnoticed.
Where does the customer stand in the "Action Cycle"?
What is the customer is trying to accomplish with your data and analysis
The customer will broadly go through six major steps in his "Action Cycle." Different locations in the cycle correspond to different types of Needs for intelligence.
Identify Agenda:

Help Customers identify key issues, trends or threats
Develop Insight:

Help customers create comm cause and joint insights around critical issues
Make Choices:

Help customers make informed, committed choices about what to do
Take Action:

Help customers take actions consistent with their choices and intended results
L
earn and Adjust:

Help customers analyze, measure, and respond to the effect of their actions
Do I understand the customers knowledge gap?
Do I understand what closing the gap would entail for analysts?
The Art of Asking Better Questions
Deduce the Customer Need
What doesn't your customer know, that they need to know?
NEED
Contextual Concerns
Action Cycle
Checking Your Analysis and Preparing Your Report
Now that you've completed your analysis, you will want to check your work to make sure you have answered your Charge well. In checking your analysis, there are two main perspectives to consider: inward-looking (how well-reasoned is my report?) and outward-looking ( how will the customer read my report?).
Understanding misalignment: examining the customer's assumptions
Often, the customer's assumptions will lead them to issue a Charge that is not well-matched to their Need.

Assumptions can most often be categorized as contextual, behavioral, or value-based:
Contextual:
beliefs about the path of the world
Behavioral:
beliefs about the motivation for human behavior, what people will do, and the lines people will and will not cross.
Value-based:

fundamental beliefs about the nature of the physical world, morality, politics, spirituality.
There is not a blueprint that applies to all cases, however these examples are meant to be indicative of how an analyst could decompose their questions in a given case.

Questions about scope and measurement:
What is the appropriate scope for my analysis?
What are the known and unknowns?
What could be the significance of outliers in my results?
Questions about underlying hypotheses:
What do I know about the relevant cause-effect relationships?
Is the customer more interested in one hypothesis than others?
Which would the customer consider good vs. evil?

What type of Charge is my Implicit Charge?
There are eight different Charge types, which can be grouped into three broad categories. It is important note that these Charge types are not mutually exclusive, since a single Charge can easily contain elements of several Charge types.

Different Charge types call for different approaches to analysis, so knowing your Charge type can help you devise an appropriate analytic strategy.
Tradecraft Tips: Charge type identification
Decompose your Questions
Decompose your questions into a set of
'answerable analytic questions'.

If a questions can be answered by applying some function to a set of data, then that questions can be considered a "terminal" question - in other words, it needs no further decomposition.

If a question cannot be answered by applying a function to a set of date, you need to further decompose that question until you reach a set of terminal questions.
Analytic Functions
The use of a specific function will depend on the data, the questions, and the relevance to the problem. Thinking through the five major categories of functions will help you decide which methods will be most useful, moreover these methods can, and often should, be used in combination
Analytic Bias
Prior to diving into the analysis itself, take a moment to think of potential analytical biases to watch for in the process. Not all biases cause problems, but it is important to be aware of them as you conduct your analysis.
Charge Type Conversion
Charge types are not mutually exclusive with analytic functions, e.g., a charge may have elements of two or more Charge types embedded in it. In many cases, combining a series of analytic functions will provide the most powerful analysis.
Analytic Bias when choosing data
Data quality bias:
when we assume that the data we have is a certain quality (inters of credibility or accuracy) without closely examining its condition
Confirmation bias:
when we search for and assign more weight to data that supports what we think or believe, then ignore or underweight data that contradicts it
Information bias:
we we believe that more information and more data is always better, more helpful, and worth knowing
Recency effect:
when we favor and assign disproportionate weight to information about recent events and experiences
Collection bias:
when we think in terms of the databases we have at our fingertips rather than the entire universe of possible data
Analytic bias when forming a hypothesis
Perspective bias:
when we phrase our question or our hypothesis in such a way that is skews our analysis in a particular direction
Worst-case scenario bias:
when we fixate on worst-case scenarios and how to fix them, at the exclusion of other outcomes
Neglect of probability bias:
when we disregard probability when making a decision under uncertainty; when we tend to favor the worst-case scenario
Disconfirmation bias:
when we pay more attention to the analysis of information that contradict what we think or believe, while simply accepting confirming information
How well-reasoned is my report?
There are many ways in which analytic reasoning can fall prey to fallacies that undermine logic.
Common logical fallacies of proof and significance
Aesthetic fallacy
:
focusing on the most clear, beautiful data, or the data that can be built most easily into a clear story
Furtive fallacy:
Assuming that things are never what they seem, and that "real" facts of special significance are secret ones
Quantitative fallacy:
assuming that "facts which count best count most," and that the more quantifiable a fact is, the greater its significance
Irrelevant proof:
Posing one question, but answering another
Negative proof:
Attempting to prove something only be negative evidence:
Possible proof:
Trying to prove the truth or falsity of a statement by establishing the possibility of its truth or falsehood
Prevalent proof:
Taking mass opinion as verification
Misplaced precision:
Seeking exactness at the cost of usefulness
Appeal to authority:
Taking expert opinion as definitive evidence
Common logical fallacies of causality
Fallacy of post hoc, propter hoc:
Assuming that if event Y occurred after event X, event X caused event Y to happen
Fallacy of cum hoc, propter hoc:
Assuming that if X and Y always occur together, one must be the cause of the other
Reductive fallacy:
Simplifying and distorting cause-and-effect in complex situations to such an extent that a single causal chain is stretched across a huge swath of data
Fallacy of indiscriminate pluralism:
Attempting to tell "comprehensive truths" and avoid the reductive fallacy by covering as many possibilities as possible and declining to prioritize any of them
Fallacy of identity:
Assuming that causes and effects must resemble each other in some way: "big effects must have big causes," and "big events must have big consequences"
Fallacy of responsibility:
Confusing and conflating agency and responsibility by demanding a single answer to the questions of "how did it happen?" and "who is to blame?"
How can I minimize the customer's misinterpretation of the report
Your report must be intelligible to accomplish its purpose, not only to your immediate customer but also to any other users or groups of users who might need its contents. Consider the users topical and analytical literacy to mitigate the risk the user will draw incorrect conclusions.
Topical literacy: sample checklist
What groups of users would be most familiar with this topic? Have I addressed their most likely follow-on questions where appropriate?
What groups of likely users might have a more difficult time of understanding this topic? What do they have to know to make sense of this document?
What related topics tend to be confused or conflated with this topic?
Has this situation arisen before? What is different this time and what is the same?
How does this topic relate to other pressing security concerns?
Analytical literacy: sample checklist
How common, widespread, or accepted is this analytical technique?
How often is this analytical technique accurately and adequately understood?
Is my approach to this data unique in any way?
Is the form in which I receive my data unique in any way?
What are the boundaries of interpretation and confidence warranted by this method?
Was my choice of analytical method unusual in any way?
Tradecraft Tips: Conversations to Explore Customer Need
Identifying the "Critical Few" Questions
What do you need to know an what would you like to know?
What gaps in your understanding of the background and context prevent you from understanding the customer's Need?
Given your sense of the relationship you have with this customer, do you expect to be able to ask questions just briefly or in depth?
Delivery
Start with questions that are Open at first, e.g.,
"I'd really like to know more about..."
Next move to Guiding questions, e.g.,
"In what way..." "What else would be helpful for me to know about X?"
Use a narrative or visual frame for communication by asking your customer to "enact" their Need, e.g.,
"I [the customer] am scheduled to give a briefing on X, and I'd like to talk about Y."
Or,
"If you had a perfect Report, what would the key charts or tables be?"
(not,
"what would they say?"
)
Play devil's advocate to test your thinking and bring your assumptions into the open.
Anticipate
"gear-shifting"
questions that would dramatically shift the direction of the conversation to a different topic, or to do a different frame on the topic.
Live Calibration
Warn the customer that this conversation may require more time, be less structured, and deal more with background and context than usual.
If your Open and Guiding questions result in the conversation becoming side-tracked, use Closing questions (including
"Who," "What," "When," "Where," "Do you," "Is,"
) that ask for description and definition to get back on track, e.g.,
"What is your definitions of 'X'?" "Do you want me to Y?" "Is this Z?"
Try to gauge your customers comfort level, and note any omissions in the dialogue.
Full transcript