What happens when the decisions we make before analysis change the evidence we eventually present?
Imagine a researcher who has completed a survey of university students. The responses have been collected, the spreadsheet is ready, and the next step appears obvious: begin the analysis.
But the dataset contains several problems.
Some participants have left questions unanswered. A few records appear twice. One student reports studying thirty hours in a week, while most others report between two and six hours. Another response contains an unfamiliar category that does not match the questionnaire’s coding scheme.
The researcher must decide what to do.
Should incomplete responses be excluded? Should the unusually high value be removed? Should duplicate-looking records be deleted? Should an unfamiliar category be recoded?
These may look like routine technical tasks. Yet every decision has the potential to change the dataset, the results and ultimately the conclusions.
Data cleaning is not merely preparation for research analysis. It is itself a series of research decisions.
1. The misconception: clean data means removing anything unusual
Data cleaning is often introduced as a simple process: identify errors, remove duplicates, fill missing values and eliminate outliers.
This description is convenient, but incomplete.
It creates the impression that there is one objectively correct, perfectly clean dataset waiting to be uncovered.
In practice, researchers frequently encounter situations in which the appropriate decision is not immediately obvious.
A missing response may indicate that a participant overlooked a question, refused to answer it, or was never eligible to answer it.
An unusually high observation may be a typing error—or a genuine observation that reveals something important about the population.
Two identical-looking responses may indicate duplication, but they may also represent two different participants who happened to provide the same answers.
The central challenge is therefore not simply identifying unusual records.
It is distinguishing demonstrable errors from legitimate variation, uncertainty and missing information.
That distinction requires methodological judgment.
2. A worked example: how cleaning choices can change results
Consider a hypothetical study exploring the weekly study hours of six students.
| Student | Reported weekly study hours |
|---|---|
| A | 2 |
| B | 3 |
| C | 4 |
| D | 5 |
| E | 6 |
| F | 30 |
Five students report between two and six hours. The sixth reports thirty hours.
The researcher notices the difference and considers removing the unusual observation.
Approach A: Keep all observations
The mean weekly study time is:
(2 + 3 + 4 + 5 + 6 + 30) ÷ 6 = 8.33 hours
The median is 4.5 hours.
Approach B: Remove the thirty-hour observation
The mean becomes:
(2 + 3 + 4 + 5 + 6) ÷ 5 = 4 hours
The median becomes 4 hours.
The difference is substantial.
| Decision | Mean study hours | Students included |
|---|---|---|
| Retain all observations | 8.33 | 6 |
| Exclude the highest observation | 4.00 | 5 |
Does this demonstrate that the thirty-hour observation should be removed?
No.
The thirty-hour value is unusual, but not impossible. It may represent an exceptionally committed student, a misunderstanding of the question, or an entry error.
The researcher should investigate its plausibility, examine the questionnaire, consider the relevant context and, where possible and permissible, verify the original response.
If the value cannot be verified, its treatment requires a justified analytical decision.
The two calculations can also be reported as a sensitivity analysis, demonstrating how much the result depends on the observation.
This example is deliberately small and hypothetical. It illustrates sensitivity to data-handling decisions, not a statistical rule for deleting observations.
Research Note
An outlier is a signal to investigate, not automatic permission to delete.
If a researcher removes an observation simply because it makes the results less convenient, the cleaning process can become a source of bias.
3. Missing data are not all the same
Missing information presents a different problem.
Imagine a questionnaire asking students about academic stress. Some participants skip the question about family income.
The researcher could remove those participants, replace the missing values, or use a statistical method that accounts for incomplete data.
But which approach is appropriate?
That depends partly on why the responses are missing.
If students with particular financial circumstances are more reluctant to disclose income, the missing responses may not be representative of those who answered.
Removing incomplete records could therefore distort the composition of the sample.
Similarly, replacing missing observations with an average may reduce apparent variability or introduce other distortions.
Researchers should examine the extent and pattern of missingness and choose a method consistent with the study design, assumptions and intended analysis.
No single missing-data technique is appropriate for every dataset.
The methodological literature emphasises planning, justification and transparent reporting of missing-data handling rather than treating it as a mechanical correction.
4. Duplicates, inconsistencies and coding errors also require judgment
Some cleaning decisions are more straightforward than others, but even seemingly simple cases benefit from investigation.
Duplicate records
Suppose two survey submissions share the same participant identification number.
If the study assigns unique identifiers, this is evidence of a potential duplicate.
However, before deleting a record, the researcher should examine timestamps, submission procedures and any authorised source records.
Identical answers alone do not establish duplication.
Inconsistent categories
A dataset records educational qualification using the categories:
- Undergraduate
- Postgraduate
- Doctoral
But some entries contain abbreviations such as UG, PG and PhD.
These may be standardised when their meanings are confirmed. The recoding rule should be applied consistently and documented.
An unfamiliar category, however, should not be forced into an existing group without justification.
Impossible values
A survey records a percentage as 850%.
If the variable is genuinely defined on a zero-to-100 scale, the value is invalid.
But identifying an invalid observation does not tell us what the correct value should be.
It may require checking the source, correcting a confirmed entry error, or marking the observation as unresolved.
The distinction between identifying an error and inventing a correction is essential.
5. Data cleaning can introduce bias
Researchers usually clean data to improve quality.
But cleaning can also damage research quality when decisions are inconsistent, unjustified or influenced by desired results.
Consider a researcher investigating the relationship between working hours and academic performance.
Students who work exceptionally long hours might have lower academic scores.
If those students are excluded simply because their working hours appear unusual, the analysis may underrepresent precisely the group relevant to the research question.
The resulting dataset could look more orderly while becoming less representative of the population being studied.
Similarly, deleting incomplete responses may disproportionately exclude participants from particular groups.
A cleaner-looking spreadsheet does not necessarily produce more credible evidence.
The appropriate goal is not to eliminate every irregularity.
It is to produce an analytically defensible dataset while preserving the integrity of the underlying observations.
6. A practical framework: detect, investigate, decide, document
A responsible cleaning process can be organised around four questions.
Step 1 — Detect: What appears unusual?
Inspect the dataset systematically.
Look for missing observations, unexpected values, inconsistent formats, questionable duplicates, coding inconsistencies and values outside defined ranges.
At this stage, record potential issues without automatically changing them.
Step 2 — Investigate: Is it an error or a legitimate observation?
Consult the questionnaire, data dictionary, collection procedures and relevant contextual information.
Ask whether the unusual observation is impossible, implausible or merely uncommon.
Distinguish verified errors from unresolved uncertainty.
Step 3 — Decide: What treatment is justified?
Choose an appropriate response based on the research question, study design, available evidence and analytical requirements.
Possible actions include correcting a verified entry error, standardising confirmed categories, retaining a legitimate extreme value, applying an appropriate missing-data method or excluding a record under defensible criteria.
Where reasonable alternatives exist, assess whether conclusions change under those alternatives.
Step 4 — Document: Can another researcher reconstruct the decision?
Record what was detected, why it was considered problematic, how it was investigated, what action was taken and how the decision affected the dataset.
Retain the original data securely, subject to applicable ethical and data-management requirements.
Where practical, use reproducible scripts or transformation logs instead of making undocumented manual changes.
SRS Research Framework
Detect → Investigate → Decide → Document
A defensible data-cleaning process should allow another qualified researcher to understand not only what changed, but why it changed.
7. What should researchers report?
A methods section that merely states, “The data were cleaned before analysis,” provides very little information.
A more informative description explains the procedures used and their consequences.
For example:
“The dataset was screened for duplicate identifiers, missing observations, invalid values and inconsistencies in variable coding. Potential anomalies were checked against the original data records where available. Confirmed errors were corrected, while unresolved observations were handled according to documented decision rules. The effects of excluding influential observations were examined through sensitivity analyses.”
This is an illustrative reporting example, not a substitute for reporting the actual methods used in a particular study.
Depending on the study, researchers should also report the number of records affected, the treatment of missing values, the criteria for exclusions and whether alternative decisions materially changed the conclusions.
Transparent reporting makes it easier for readers to evaluate the reliability of the findings.
8. The larger lesson: data are not neutral once we start transforming them
Research data do not move directly from collection to conclusions.
Between those stages, researchers classify, correct, recode, exclude, transform and summarise observations.
Each operation may involve assumptions about what counts as an error, what counts as valid evidence and which differences matter.
This is why data cleaning belongs within the logic of research methodology rather than being treated as an invisible technical stage.
The researcher must be able to defend the decisions made before statistical analysis just as carefully as the analytical techniques chosen afterward.
A good research study does not require a dataset without imperfections.
It requires an honest and systematic account of how those imperfections were investigated and managed.
The quality of research evidence depends not only on what the data contain, but also on what researchers decide to do with them.
Further reading
Van den Broeck, J., Argeseanu Cunningham, S., Eeckels, R., & Herbst, K. (2005). Data cleaning: Detecting, diagnosing, and editing data abnormalities. PLOS Medicine, 2(10), e267. https://doi.org/10.1371/journal.pmed.0020267
Bakker, M., & Wicherts, J. M. (2014). Outlier removal and the relation with reporting errors and quality of psychological research. PLOS ONE, 9(7), e103360. https://doi.org/10.1371/journal.pone.0103360
National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and Replicability in Science. National Academies Press. https://doi.org/10.17226/25303