How we read a conversation

You can’t measure leadership directly, the way you can measure temperature. You have to break it into behaviors you can observe, and then show that those behaviors add up to something real. Most attempts to measure human behavior go wrong at that second step.

They go wrong in one of two ways. Some measure a thing very consistently that turns out to have little to do with leadership, which is a problem of validity. Others pick out something real but measure it so erratically that the numbers are mostly noise, which is a problem of reliability. The method is designed to guard against both.

Feedback research gives a third reason for caution. In their 1996 meta-analysis of more than 600 effects from feedback studies, Kluger and DeNisi found that over a third of feedback interventions left performance worse than no feedback at all. Most of the damage came from feedback that pulled attention away from the task and onto the person, which is what vague feedback does. That finding has shaped Parafoil more than any other.

No single signal decides anything. Each layer answers a different question, and a reading from one layer only reaches you if the others support it.

The layers combine conjunctively. Any claim about a specific moment needs the supporting layers to agree, and when they disagree the claim is dropped instead of averaged. A strong reading on one layer cannot make up for a weak one on another.

The first layer is the least intuitive of the eight. The example below takes one sentence, fades its content words and underlines its function words.

Without the content words the topic is gone, but the structure is still there: who the subject of the sentence is, whether a decision is being announced or offered, and where the hedging sits and at whom it is aimed. One sentence proves nothing on its own, but across several hundred turns these patterns are stable enough to say something about how you lead.

Any system like this risks building in somebody’s private opinion of how people ought to behave. We guard against that in three ways.

Claims come from published research. When the product says a behavior matters, that comes from the behavioral and management literature and is cited internally, which keeps our own preferences out of it.

You’re compared with your own organization and your own history. The reference points come from how leadership works where you work and from how you led last month. Being more direct than average means little until you know what average looks like in your setting.

You set the target. At onboarding you choose what you want to work on, and the guidance is weighted toward the kind of leader you said you want to be. That makes Parafoil closer to an instrument you point at your own goals than an assessment.

The weighting changes what is surfaced to you. It never changes the measurement underneath, so two managers who behave identically are scored identically, even if they are shown different things. This is what keeps organization-level views comparable across people who chose different goals.

The reasonable suspicion about any language system is that it produces plausible prose with nothing behind it. Three safeguards address that.

Everything is scored against a human reference. The primitives are measured against the expert panel’s judgments on the same material, so the system never marks its own work, and people can show when it is wrong.

It abstains. If the audio is poor, if speakers cannot be reliably told apart, or if your own words are badly transcribed, you get no score. Abstention is a designed output with its own criteria, because no reading is better than a confident wrong one.

Every claim is anchored. When Parafoil says something happened, it shows you the moment and the words, so you can check the observation against what was actually said.

Agreement between expert raters can be measured, and there is a standard toolkit for doing it: chance-corrected agreement statistics for categorical judgments (Fleiss, 1971), intraclass correlation for ratings on a scale (Shrout & Fleiss, 1979), and confidence intervals estimated by resampling, so that no single figure is read on its own. The same methods are used to validate clinical and psychological instruments.

When experienced managers independently reach the same reading of the same moment, the behavior is defined well enough to score. When their ratings scatter, the problem is usually the definition, so we revise it and hold the number back.

Each behavior therefore sits on a ladder, and its position determines what the product may do with it:

1. LowNot used. The definition needs revising before the measurement means anything.
2. ModerateUsed only for aggregate patterns, which hold across many observations even when single observations are unreliable.
3. GoodValidated for analysis and for trends over time.
4. High — the highest barUsed for individual feedback. Only at this level can a behavior drive feedback about one specific person.

The bar for saying something about you is the highest one. A behavior below it can contribute to a pattern across hundreds of meetings, but it is not used to tell you what you did in a particular conversation.

Small panels produce wide confidence intervals, sometimes wide enough to span two rungs. When that happens, the lower rung applies until more raters or a sharper definition narrows the interval.

How a behavior earns its place

Every primitive starts in the literature. It is defined in writing, checked against the research it came from, and frozen before any conversation is scored against it. The panel then tests whether experienced managers reading the same moment reach the same judgment.

A behavior has to pass both tests before the product uses it, and it is retested as the panel grows. One that stops holding up moves back down the ladder.

A system that reads how people talk will reward some ways of talking over others. We would rather set out that risk here than wait to be asked about it.

The risk is greatest for three groups: people working in a second language, people from cultures where directness and deference work differently, and quiet people who contribute a great deal while talking little.

What we do about it

• The panel is built for range. Raters differ by function, sector, region, gender and ethnicity as well as by management background. A reference standard inherits the blind spots of the people who set it, so the make-up of the panel is part of the measurement.
• Comparison stays local. Measuring you against your own organization and your own history avoids much of the cultural penalty that a single global ideal would impose.
• We score the outcome, whatever the route. What we measure is whether ownership of a decision actually moved, and very different styles that achieve it should score the same.
• Speaking time is context. It describes the meeting and earns no credit on its own.
• Coverage is monitored. We track, for each person, how often the system declines to score a conversation. A high rate is treated as an audio or transcription fault and fixed, so that nobody ends up with thinner feedback than their peers.

Language and second-language speakers

The principle carries across languages, but the calibration has to be built for each one. Languages that routinely drop subject pronouns, or that encode status in honorifics, need their own mapping before the same inferences hold.

Second-language speakers are the case we watch most closely. Someone working in a shared company language can show patterns that come from switching languages and have nothing to do with how they lead, so those readings are held to a higher bar before anything is said about them.

We monitor scores for systematic differences between groups, and we would rather hear about a problem from you than miss it. If a reading seems to reflect how someone speaks and not how they lead, please tell us.

Parafoil starts with what happens in the meeting. It won’t know that you settled a question over Slack that morning, or that the person you were short with had asked you to be direct, so when a reading feels wrong, missing context is usually why, and you can set it aside.

A single conversation is a small sample, and often from an unusual day. This is why Parafoil learns about you over time. The longitudinal value of Parafoil is a developed understanding of who you are, how you behave and what is changing, considered alongside any context you give it at the start.

That context keeps growing. What you write in the About me section of your settings, and what comes up in your conversations, teaches Parafoil about the work you are doing, so its guidance fits you more closely the longer you use it. When it names a style, it is describing a pattern in how you behave, and it makes no judgment about what kind of manager you are.

Almost everything a manager learns about how they lead arrives as a survey, a review or a colleague’s summary, usually months after the fact. Each of these is a recollection of what happened, collected once and compressed into a form.

The limits of recollection are well documented. People remember the recent and the dramatic and lose the ordinary middle, which is where most managing happens. What colleagues are willing to write down differs from what they noticed, and in multi-rater feedback a substantial share of the variance reflects the person giving the rating as much as the person being rated. The feedback also tends to arrive once a year, covering twelve months of behavior.

This is where the feedback research matters most. Feedback that draws attention to the person instead of the task is the kind most likely to make performance worse (Kluger & DeNisi, 1996), and “be more strategic”, arriving nine months after the meeting that prompted it, is about as vague as feedback gets.

Parafoil reads each conversation shortly after it happens, and it reads every one you choose to include, so the ordinary middle is the part it sees most of.

On coaching

Coaching and Parafoil work well together. What Parafoil adds is breadth: one person reads you through their own training, experience and mood on the day, and a reference built from many experienced managers spreads those influences across all of them.

Each of the following is technically within reach, and we have decided against it.

We don’t assess anyone but you

Parafoil analyzes whole conversations, so other people’s words are part of the material, but they are never its subject. Nobody else in your meetings gets a score, a profile or a reading of their own, and no view anywhere in the product produces one. What is measured is how you lead, and everyone else in the room is context for that.

We don’t read faces

Video analysis of facial expression and posture is commercially available and some products use it, but the evidence behind it is weaker than the enthusiasm for it. A 2019 review by Barrett and colleagues concluded that emotional states cannot be reliably inferred from facial movements alone. A measurement we could not defend would be worse than none, so we will not use it unless the science changes.

We don’t read your email or your chat

Parafoil analyzes the meetings you choose, and only the meetings you choose. It doesn’t analyze email or chat, because how you cognitively process those is different. Parafoil’s job is to improve the experience of working with you, to make you a more effective leader.

You don’t have to take our word for the science. The work below is published, and most of it is the standard reference in its field. Pennebaker’s book is the best place for a general reader to start, and the rest is the academic literature the layers are built on.

Each title links to a literature search for that work, so you land on the paper itself and on everything that has cited it since.