Reliability vs validity is the foundational distinction in psychology research methods: reliability is about consistency — whether a measure produces stable, repeatable results — while validity is about accuracy, whether it actually measures what it claims to measure. Every psychometrics question, from interpreting a scale's Cronbach's alpha to criticising a study's questionnaire, rests on knowing which of the two you are being asked about.

This guide gives you the four flavours of reliability, the four flavours of validity, the bullseye logic that explains why a test can be reliable but not valid (and never the reverse), a worked application to a real screening scale, and a four-step answer template that scores marks in essay and MCQ formats. If you are revising for a psychology or psychiatry exam, pair it with Preppsy's general psychology exam track for the surrounding research-methods content.

Reliability vs validity: the core difference

Reliability = consistency. Ask: if this were repeated — same person, later date; different rater; parallel form — would the score look the same? A reliable measure has low random measurement error; its scatter is tight, whatever the target.

Validity = accuracy. Ask: does the score mean what we say it means? A valid measure's scatter is centred on the truth. The classic exam formulation: reliability concerns the consistency of the measuring instrument, validity concerns the meaning of the scores.

The relationship is asymmetric and exam-critical: reliability is necessary but not sufficient for validity. You cannot have validity without reliability, but you can absolutely have reliability without validity.

Types of reliability

Test-retest reliability

Same test, same people, two occasions. It checks temporal stability. Threat: genuine change between occasions (or practice effects), so an unstable construct like mood will look "unreliable" for sensible reasons. Typically summarised as a correlation between occasion 1 and occasion 2 scores.

Inter-rater (inter-examiner) reliability

Two or more raters score the same material. Central for structured interviews and MSE-style ratings, where drift between raters destroys comparability. Reported as percentage agreement or chance-corrected coefficients such as Cohen's kappa.

Internal consistency

Do the items within one instrument hang together? The workhorse statistic is Cronbach's alpha. The commonly used — but conventional, not absolute — threshold is alpha ≥ 0.70 for research use, with 0.80+ preferred for individual-level decisions; interpret it hedged, because alpha depends on item count and assumes items measure one construct (tau-equivalence). Split-half and alpha-if-item-deleted diagnostics are the supporting cast.

Alternate-forms reliability

Two parallel versions of a test given to the same people — the practical answer to test-retest's practice-effect problem, and the reason exams commission multiple question banks.

Types of validity

Content validity

Do the items represent the full domain? Judged by expert panels and blueprints — the logic behind every exam's specification document.

Construct validity

Does it measure the theoretical construct? Established cumulatively through convergent validity (correlates with measures of related constructs) and discriminant validity (does not correlate with unrelated ones), plus factor structure. If a questionnaire's factor analysis loads as theorised and it correlates moderately with sibling constructs but weakly with unrelated ones, construct validity accumulates.

Criterion validity

How well does it predict or match an external criterion? Concurrent = validated against the criterion now (e.g. against a diagnostic interview at the same sitting); predictive = predicts the future (baseline score forecasting later outcome).

Face validity

Whether the test merely looks like it measures the construct. Weak scientifically, but it drives acceptability and engagement — a questionnaire that seems irrelevant to respondents invites careless answers.

The bullseye: why reliable does not mean valid

Picture a target. Reliability is grouping; validity is placement. Tight clusters far from the bullseye = reliable but not valid: the instrument repeats the same wrong answer with precision — a bathroom scale systematically miscalibrated by 3 kg, or a questionnaire that consistently measures respondent anxiety rather than the depression it claims to capture. Systematic (constant) error shifts the group without adding scatter, so consistency survives while accuracy dies.

The reverse — valid but unreliable — is logically impossible: a measure that misses the truth inconsistently has no defensible claim to ever hit it. Random measurement error widens the scatter, and a widely scattered measure cannot be said to have captured the construct on any given occasion. In short: reliability is a prerequisite for validity, never a consequence of it.

DimensionReliabilityValidity
Core questionIs the measure consistent?Does the measure measure what it claims?
Threatened byRandom measurement errorSystematic error and poor construct mapping
Common formsTest-retest, inter-rater, internal consistency (Cronbach's alpha), alternate formsContent, construct (convergent/discriminant), criterion (concurrent/predictive), face
Typical evidenceCorrelations and coefficients between occasions, raters or itemsExpert panels, factor analysis, correlations with established measures or outcomes
Can it stand alone?Yes — but only says results repeatNo — requires reliability first; validity without consistency is incoherent
Bullseye analogyShots grouped togetherShots centred on the bullseye

Exam trap

"A test can be reliable but not valid" — TRUE. Explain it with the bullseye: tight grouping away from the centre (systematic error preserves consistency). "A test can be valid but not reliable" — FALSE. If scores scatter randomly they cannot consistently hit the truth, so validity cannot be claimed. State the asymmetry explicitly: reliability is necessary but not sufficient for validity.

Worked example: reliability and validity in a real scale

Take a depression screener such as the PHQ-9. Reliability evidence: internal consistency — items about mood, sleep, appetite and concentration cohere, with reported alphas comfortably above the 0.70 convention (check the original validation paper for exact figures), and repeated administration in a stable patient yields similar totals. Validity evidence: content validity from DSM symptom coverage, concurrent validity against a diagnostic interview, predictive validity for functional outcomes, and construct validity from expected correlations with anxiety and functioning measures.

Cultural and translation validity is the angle regional exams favour: translating or adapting a scale for, say, an Indian or Australian sample is not a purely linguistic exercise. Items must retain meaning locally — "loss of interest" idioms, somatic versus psychological expression of distress, dietary items that assume local foods — otherwise content and construct validity degrade even though the coefficient stays high. Local validation studies exist precisely because reliability in the original language says nothing about validity in the new one.

The exam answer template for reliability vs validity questions

Short-answer and essay marks cluster around four moves. Use them every time:

  1. Define. Reliability = consistency of measurement; validity = accuracy / measuring what is claimed.
  2. Name the types. Reliability: test-retest, inter-rater, internal consistency (Cronbach's alpha), alternate forms. Validity: content, construct (convergent, discriminant), criterion (concurrent, predictive), face.
  3. Give an example. Two raters scoring the same interview (inter-rater); a screening questionnaire correlating with a diagnostic interview (concurrent validity).
  4. State the relationship. Reliability is necessary but not sufficient for validity; reliable-but-not-valid is possible (bullseye), valid-but-not-reliable is not.

For MCQ stems, watch the direction of the claim: an item describing repeatable scores is probing reliability; an item describing whether the test captures depression, intelligence or the target construct is probing validity. And when a stem quotes "Cronbach's alpha ≥ 0.70", treat it as a widely used working convention — a floor for group-level research, not a law of nature, with higher thresholds expected for high-stakes individual decisions.

Key takeaways

  • Reliability = consistency across time, raters or forms; validity = measuring what it claims to measure.
  • Four reliability types (test-retest, inter-rater, internal consistency, alternate forms) and four validity types (content, construct, criterion, face).
  • Bullseye logic: reliable-but-not-valid is possible via systematic error; valid-but-not-reliable is impossible — reliability is necessary but not sufficient for validity.
  • Cronbach's alpha ≥ 0.70 is a common convention, hedged: it depends on item count and unidimensionality assumptions.
  • Translation and cultural adaptation threaten validity even when reliability coefficients stay high — always check local validation evidence.
  • Answer structure: define → name types → give example → state the relationship.

Drill the definitions and the bullseye logic with Preppsy's spaced-repetition flashcards so the distinction is instant under exam pressure.

Frequently asked questions

What is the difference between reliability and validity?

Reliability is consistency — whether the same measure gives stable, repeatable results across time, raters or forms. Validity is accuracy — whether the measure actually assesses the construct it claims to. A reliable instrument produces the same answer repeatedly; a valid instrument produces the right one.

Can a test be reliable but not valid?

Yes. In bullseye terms, scores can group tightly away from the target: systematic error (for example, a miscalibrated scale or a questionnaire tapping the wrong construct) preserves consistency while destroying accuracy. The reverse cannot occur — valid but unreliable is logically impossible — so reliability is necessary but not sufficient for validity.

What is Cronbach's alpha and what value is acceptable?

Cronbach's alpha is a coefficient of internal consistency: how well the items within a scale hang together. A value of 0.70 or above is the conventionally used threshold for research purposes, with 0.80+ preferred for individual-level decisions — but treat this as a working convention, since alpha rises with item count and assumes the items measure a single construct.

Why does a translated questionnaire need local validation?

Because translation can silently break content and construct validity: idioms, somatic expression of distress and culturally specific items may change meaning even when wording looks equivalent. Reliability in the original language carries over only partially, so local studies must re-establish validity evidence in the new population before scores are interpreted.

Cover image: “Reliability vs Validity (cover)” by Preppsy Editorial Team (Original artwork) via Preppsy.