Redboat Ventures
← All insights

What does 'it worked' actually mean for a communication campaign?

What does it mean when communication 'works'? Misinformation studies reveal why a positive result can leave the most important question unanswered.

Share
What does 'it worked' actually mean for a communication campaign?
A grayscale close-up of a vintage measuring instrument, partly covered by torn cream paper and halftone newsprint, with deep teal and rust-red collage accents.

Imagine a campaign video to help people become less susceptible to misinformation. The video teaches viewers how misleading claims are constructed, and how to spot them. In a follow-up test, the audience is found to be less likely to believe false headlines. It is an encouraging result, but it leaves a crucial question open: have viewers become better at recognising misinformation, or simply more sceptical of everything they read?

If the test includes only false headlines, these two outcomes could appear the same. To tell them apart, researchers also need to see what happens when people encounter accurate news. The same problem appears in communication projects more broadly: a measure can improve while the outcome we actually care about remains unknown.

Research graphic showing that, among 881 psychological studies, 61.5% used both true and false news stimuli and 36.3% used false news only. These categories do not make up a complete breakdown. Source: Webster et al. (2026).

The problem with measuring false news only

A November 2026 scoping review by Webster, Modirrousta‑Galian & Wimmer, examined the news items used in 881 psychological studies of misinformation. In 36.3% of the studies, participants saw only false news, with no genuine news for comparison. In those studies, the design can measure whether people reject false claims, but it cannot show whether they have become better at distinguishing false from true claims. [1]

Consider this illustrative test. Before watching a video, someone believes six of ten false headlines; afterwards, they believe only two. The improvement looks substantial. Yet if the video has made them distrust news generally, they might also reject eight of ten true headlines. They would be more sceptical without necessarily being better informed.

While Webster and colleagues's review doesn't claim that this happened in the studies it examined, it did identify what a false-only test cannot rule out. For a project intended to improve judgement, it matters whether people can recognise reliable information as well as reject unreliable information.

Two rows of torn photographic news fragments; one is marked with red circles while the other is partly hidden beneath translucent paper.

Exposure to false news reduced trust, increased over-confidence, but did nothing for ability to tell false from true

Researchers ran two online experiments with 2,735 people in the United States in 2026. Some people saw feeds where about one in six headlines was false; others saw feeds where as many as five in six were false. Afterward, they were asked to judge whether a set of headlines was true or false.

Greater exposure to false news reduced trust in news and increased overconfidence in participants' ability to distinguish true from false headlines. But they did not get better at telling true headlines from false ones. [2]

Redboat Insights graphic showing five false-news exposure levels tested: 17%, 33%, 50%, 66% and 83%. As exposure increased, trust in news fell while confidence and overconfidence rose; discernment did not improve. At 83% versus 17%, trust was 0.24 points lower and overconfidence was 6.11 percentage points higher. The trend lines show direction, not exact values for each level. Source: Altay, Lyons and Modirrousta-Galian (2025).

A separate survey experiment, reported in a preprint, found that people made more errors on whichever type of headline had been less common in the feed they had just seen. After a mostly true feed, they more often misidentified a subsequent false headline; after a mostly false feed, they reverse pattern appeared. The preprint offers another reason to inspect responses to both kinds of news rather than counting only the claims people reject. [3]

Together, the studies show why distrust, confidence and accuracy need separate measures. An audience can feel better equipped to judge a claim while its actual judgements tell a different story.

Grainy grayscale close-up of one eye beneath layered blue and red torn paper, suggesting the gap between confidence and clear judgement.

Decide what success would mean

The distinction extends beyond misinformation. A science animation might improve viewers' answers to questions about how a treatment works. That is useful evidence of learning, but it does not, by itself, show that viewers understand the risks well enough to make an informed decision. A sustainability campaign might attract views, shares and stated intentions without showing whether behaviour changed. These are illustrative situations, not reported campaign results.

Recall, engagement and intentions can all be valuable. The mistake would be to let one stand in for another. Watching a video, answering a question correctly straight afterwards, or feeling confident doesn’t necessarily mean someone will remember the information or use it well later.

Three questions to ask before production begins

Start with these three questions:

1. What change do we want?

Name the intended outcome precisely: remembering an instruction, understanding a process, recognising reliable evidence, making an informed decision or changing a behaviour. These are different outcomes, so decide which one(s) the content is meant to achieve. Trying to cram too many outcomes into a single piece of content can dilute it's effectiveness.

2. What will the test actually measure?

Choose measures that match the outcome. A recall question can show what someone remembers straight after viewing; understanding may need a different test, and later application may need a follow-up. Be clear about what the evidence can and cannot show.

3. What else might change?

Check for unintended effects as well as the intended outcome. If a measure improves, look at related outcomes too. That helps show whether the change truly reflects the intended improvement or comes with a trade-off.

Even when a campaign scope doesn't support detailed evaluation, these questions help choose measures that match their goals - and make clear what the results can support.

A woman studies a printed sheet at a desk, with one hand holding a pencil, as blue and red torn-paper paths diverge behind her.

Say exactly what changed

Before calling communication successful, be precise about what the evidence shows. If people remembered more, report improved recall. If more people watched or shared the content, report stronger engagement. Claim better decisions or changed behaviour only if those outcomes were measured.

Being exact doesn’t diminish a positive result. It shows what the communication achieved, where the evidence ends and what to measure next.


Sources

[1] Webster, T. C., Modirrousta-Galian, A., Skelton, F. C., & Wimmer, M. C. (2026). The news we use: A systematic scoping review of fake news stimuli in psychological research. Behavior Research Methods, 58, article 343. Published 4 November 2026.

[2] Altay, S., Lyons, B. A., & Modirrousta-Galian, A. (2025). Exposure to higher rates of false news erodes media trust and fuels overconfidence. Mass Communication and Society, 28(2), 301–325. Published online 7 August 2024.

[3] Orchinik, R., Martel, C., Rand, D. G., & Bhui, R. (2025). Adaptive intuitions shape susceptibility to misinformation. PsyArXiv preprint, version 2.

Subscribe to Redboat Insights

Get new articles and insights delivered to your inbox.

Subscribe to Redboat Insights

Get new articles and insights delivered to your inbox.

Check your inbox to confirm your subscription.