This is my personal Book of Shadows. If you find it useful or helpful in any capacity, please consider buying me a Coffee.

Research and Study Types

All research is not created equal. The best that we can do in many cases- especially as laymen with no formal professional training- is treat information with the general respect it deserves; ignoring grandiose claims and proclamations whenever made, examining the studies and evidence for ourselves whenever possible, following source trails as far as we are able, and paying attention to the peer review process (and to the real professionals speaking on the subjects)... Only in this way can we truly hope to educate ourselves effectively and with integrity.

Critical thinking is an integral part of this process; hundreds of studies and papers may be published daily- and not all of them good. Evidence may be incomplete or spotty, or hypotheses contradictory. This often leaves plenty of room for ‘creative interpretation’ of the facts: The very sort of selective reasoning that allows so many to pass their version off as ‘the definitive truth’ on the matter. Strong critical thinking skills, then, will allow you to cut through the nonsense. Understanding a few basic facts about studies, types of studies, and how they work, also helps:

1. Human Studies Trump Animal & Plant Studies

If it’s not meant to concern human physiology or behavior, animal and plant studies can sometimes provide us with valuable data. Ultimately, however, the best data comes, of course, from purely Human or Human-dominant studies.

2. Prospective Studies Trump Retrospective Studies

Collecting data about things in real time, as it’s actively occurring (prospective) logically provides much more reliable data than trying to get people to self report accurately about the past.

3. Randomized Control Trials Trump Almost Anything

The classic scientific method was developed for a reason.

4. Systematic Reviews «vs» Meta-Analysis

These are both studies of studies that are conducted in order to find broader trends. These are good, but they rely on the studies being studied to have been well performed themselves- which is often not the case.

5. Case Series & Anecdotes Are Always Worthless

Anecdotes are personal stories like “I ate a spoon full of tabasco every day for a year and I never got sick”. Case Series’ are collections of lots of Anecdotes that form a ‘study’ looking for trends. Neither actually have any real scientific use or validity.

6. Relative Risk «vs» Absolute Risk

Absolute Risk is harder to prove, but the media loves to focus on Relative Risk because it sounds much bigger and scarier than it actually is. If you want to make a truly informed decision where risk is a factor, however, then Absolute Risk is what you should actually look at.

Of course, not all types of studies will matter to Pagans. But having a general idea of some of the studies which exist and how they work, and where you should be focusing your attention (should you need to), is always helpful. This is especially true if we hope to maintain historical integrity, and can be additionally useful to those venturing onto the path of the Herbarius. Additionally, there are a few other tips that will help you:

  • Be especially skeptical about surveys and polls. Who is funding the project; how the questions are chosen, worded and posed; how those questioned are selected and the context in which the questions are put to them; how the statistical analysis is carried out and the statistics are interpreted; how the findings are presented and reported (or misreported) – all these factors can have a massive influence.

  • Have at least a basic understanding of statistics. A figure without a context is often meaningless. For instance, it might be reported that this weekend there were 10 deaths on Britain’s roads. What does this mean? To obtain some context, it would be useful to know the number of deaths for each weekend in the last month, year or decade and more helpfully calculate the average figure for each of those periods. Sometimes, however, the average can be a misleading measure; it is good to know the difference between mean, median and mode. Sometimes the median or mode may be a better indicator than the mean by reducing the impact of ‘extreme’ instances.

  • Make appropriate use of statistics particularly where subjective judgements are likely to dominate. A classic example is the notion of risk where most people have incredibly subjective perceptions. Obviously crossing a road is risky, because you might be hit by a car. But staying at home is risky too; you might fall down the stairs or electrocute yourself with the toaster or be attacked by an abusive husband; everything in life has an element of risk. What is necessary is to quantify that risk in terms which make it comparable with other risks and then take rational decisions based on reasonable probabilities of particular outcomes. This may seem a complicated manner of deciding whether to take the train or fly, but it is a technique which can be used in many other situations where emotions can blind sensible decisions, such as deciding whether to use hormone replacement therapy or where to allocate scarce resources in health care.

  • Be especially cautious of statistics where probability is concerned. Most people with no training in statistical techniques find it very difficult to make an accurate assessment of probability. For instance, assessing how many people you would need in a room before it was likely that two of them shared the same birthday: In this scenario, ‘likely’ means with over 50% probability. In fact, the answer is only 23 because we are talking about any birthday matching, not one specific birthday. See how one can so easily be misled by probability?

  • Make geographical comparisons. If the government claims that it is now spending 10% of Gross Domestic Product on the health service, compare that with the percentage expenditure in other industrialized countries. If you live in a European Union country, there are often comparative statistics or policies in the other 25 Member States. If you live in the USA, there are often comparative statistics or policies in the other 49 states.

    Main Sources