a. The scientific method: a method of research in which
- a problem is identified,
- relevant data are gathered,
- a hypothesis is formulated from these data, and
- the hypothesis is empirically tested.
Historical note: credited to al-Haytham, a scholar working in Cairo during the late 10th, early 11th century.
b. Experimental sciences: In experimental science, the “testing” step of the scientific method is often implemented by a method known as “hypothesis testing.” The method is a high intellectual achievement of the 20th century, but it does not provide universal safeguard against making errors.
The method aims at quantifying the size of an effect observed in data. The quantity encodes the significance is called p-value. Scientific journals filter out publishing experimental results with non-significant effects, accepting mainly results with p-value below a threshold (usually 5%). By designed, the method produces incorrect findings (Type I errors), p% of the times.
Consequences: PUBLICATION BIAS
- studies that find low evidence of effects are ignored
- there are studies that find strong effects by a lucky draw of observations that end-up being published
Canadian connection: Ronald Fisher, one of the main contributors to the p-value methodology, presented theory behind testing hypothesis during the 1924 International Congress of Mathematicians held in Toronto.
Fisher, R.A. (1924) On a Distribution Yielding the Error Functions of Several Well Known Statistics, Proceedings of the International Congress of Mathematics, Toronto, 2: 805-813
c. Is there something wrong with the scientific method?
“But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants”
1. Given publication bias, and the resulting wearing of observable effects, what is the prudent dose of skepticism towards scientific findings, short of being a science denier?
2. difference between “science” and “experimental research”?
3. who should be vigilant: researchers, consumers, professionals, government,..?
4. Any favorite science triumph? Blunder?