Richard Feynman once said of the scientific process, “The first principle is that you must not fool yourself — and you are the easiest person to fool.” The idea that scientists might be fooling themselves (whether out of ignorance or in order to preserve their jobs) is a common accusation made by skeptics of scientific disciplines ranging from climate change to cosmology. It’s easy to dismiss such criticism as unfounded, but it does raise an interesting question: how can we tell that we’re not fooling ourselves? 

The popular view of science is that experiments should be repeatable and falsifiable. If you have a scientific model, that model should make clear predictions, and those predictions must be testable in a way that can either validate or disprove your model. It is sometimes assumed by critics that this means the only true sciences are those that can be done in a laboratory setting, but that’s only part of the story. Observational sciences such as cosmology are also subject to this test, since new observational evidence can potentially disprove our current theories. If, for example, I observe a thousand white swans I might presume that all swans are white. But the observation of a single black swan can overturn my ideas. A scientific theory is therefore never absolute, but always tentative, depending on whatever subsequent evidence arises.

Image credit: Sergio Valle Duarte, under c.c.-by-s.a. 4.0.

Even though it’s technically correct, calling well established scientific theories “tentative” is a bit misleading. For example, Newton’s theory of universal gravity stood for centuries before being supplanted by Einstein’s theory of general relativity. While we can now say that Newtonian gravity is probably wrong, it is as valid as it ever was. We now know that Newton is an approximate model describing the gravitational interaction of masses, and it is such a good approximation that we still use it today for things like the calculation of orbital trajectories. It’s only when we extend our observations beyond the (very large) range of situations where Newton is valid that Einstein’s theory becomes necessary.

As we build a confluence of evidence to support a scientific theory, we can be confident that it’s valid with the small caveat of being open to new evidence. In other words, the theory can be considered “true” over the range for which it’s been robustly tested, but new regimes might uncover unexpected behavior that leads to an advancement, and a more complete picture. Our scientific theories are intrinsically tentative, but not so tentative that we can’t rely upon their accuracy. It seems a reasonable position, but it raises a challenge for well established theories. Since we can never know for sure that our experimental results are the “real” results, how can we be sure we aren’t simply reinforcing the answer we expect?

Recommended speed of light values over time. Adapted from Henrion & Fischhoff (1986).

This line of thinking comes up a lot in introductory physics courses. Students are assigned to measure some experimental value such as the acceleration of gravity or the wavelength of a laser. Being novice experimenters, they sometimes make basic mistakes and get a result that doesn’t agree with the “accepted” value. When that happens, they’ll go back and check their work to find a mistake. However if they make mistakes in such a way that their errors either cancel out or aren’t apparent, they won’t tend to double check their work. Since their result is close to the expected value, they assume they must have done things correctly. This confirmation bias is something we all have, and can happen with the most experienced researchers. Historically this has been seen with things like the charge of an electron or the speed of light, where the initial experimental results were a bit off, and subsequent values tended to agree with earlier results more than the current values.

Timeline of the universe. Image Credit: NASA/WMAP Science Team, modified by Ryan Kaldari.

Currently, in cosmology, we have a model that agrees very strongly with observational results. It’s known as the ΛCDM model, so-named because it includes dark energy, represented by the Greek letter Lambda (Λ), and cold dark matter (CDM). Much of the refinement of this model involves making better measurements of certain parameters in this model, such as the age of the universe, the Hubble parameter, and the dark matter density. If the ΛCDM model is indeed an accurate description of the Universe, then an unbiased measurement of these parameters should follow a statistical pattern. By studying the historical values of these parameters, we can determine whether there is bias in the measurements.

Image credit: Wikimedia Commons user Dan Kernler.

To see how this works, imagine a dozen students measuring the length of a chalkboard. Statistically, some students should get a larger or smaller value than the real value. Following a normal distribution, if the real value is 183 centimeters with a standard deviation of a centimeter, then one would expect about 8 of the students to get a result between 182 – 184 centimeters. But suppose all of the students were within this range. Then you might suspect some bias in the results. For example, the students might figure the chalkboard is likely 6 feet wide (182.88 centimeters), so they make their measurement expecting to get 183 centimeters. Paradoxically, if their experimental results are too good, that would lead you to suspect an underlying bias in the experiment.

In cosmology, the various parameters are well known. So when a team of researchers undertake a new experiment, they already know what the accepted result is. So are results being biased by prior results? A recent work in the Quarterly Physics Review looks at this very question. Looking at 637 measurements of 12 different cosmological parameters, they examined how the results were statistically distributed. Since the “real” values of these parameters is not known, the authors treated the WMAP 7 results as the “true” values. What they found was that the distribution of results was a bit more accurate than they should be. It wasn’t a huge effect, so it could be due to an expectation bias, but it was also significantly different from the expected effect, which could mean there was an overestimation of the experimental uncertainties. It also meant, when the 2013 Planck data came in, the “shift” in the parameters was somewhat outside the range that most cosmologists had measured.

Image credit: Planck collaboration / P.A.R. Ade et al. (2013), annotations by E. Siegel.

This doesn’t mean our current cosmological model is wrong, but it does mean we should be a bit cautious about our confidence in the level of accuracy of our cosmological parameters. Fortunately, there are ways we can determine whether this anomaly is due to some amount of bias, such as doing blind analysis or encouraging more open data, where other teams can do a re-analysis using their own methods and the same raw data. What this new work shows is that while cosmologists aren’t fooling themselves, there is still room for refinement and improvement of the data, methods and analyses they undertake.

Paper: Croft, Rupert A.C. et al. On the measurement of cosmological parameters. Quarterly Physics Review (2015) No 1 pp 1-14 arXiv:1112.3108 [astro-ph.CO]

This post first appeared on Starts With A Bang!

The post Are Cosmologists Fooling Themselves? appeared first on One Universe at a Time.

The image above shows a pair of colliding spiral galaxies known as Arp 274. What’s interesting is not that they happen to be colliding, but that the two galaxies are spiraling in opposite directions. The one on the left spirals in a clockwise direction, while the one on the right spins in a counterclockwise direction. Sometimes we’ll refer to the left galaxy as left-handed, while the right one is right-handed. The reason is that if you hold hands up with your thumbs pointing at yourself, you’ll see the fingers on your left hand curve clockwise, and the fingers on your right hand curve counterclockwise.

These two galaxies just happen to spiral in opposite directions, but it raises an interesting question. Is there a statistical bias to the orientation of spiral galaxies? That is, are there more left handed spiral galaxies than right handed ones, or vice versa? Intuitively you would think not. If galaxies are oriented in random directions, then we’d expect to see an even mix of right and left handed spiral galaxies (when we can tell a direction). Still, with crowdsourced science it is a fairly straightforward thing to test, and this is exactly what was done over at Galaxy Zoo.

What the team did was take images of about 35,000 spiral galaxies, and had volunteers determine whether each one was right-handed or left-handed. They made sure each galaxy was evaluated by multiple people, and only took a direction as “determined” if 4 out of 5 volunteers agreed on a particular direction. What they found was of the clearly decided spiral galaxies, about 52% were right-handed, and 48% left-handed. This is very clearly a bias toward right-handed galaxies. The chance of that happening purely by chance is about 1 in 400 trillion.

That might seem like clear evidence of some inherent handedness to spiral galaxies, but you have to be careful about drawing conclusions too quickly. One of the well known phenomena in psychology is the fact that people have a bias towards right-handedness. Whether it is innate or culturally driven is not clear, but in general we tend to prefer right-handed things to left-handed ones. Of course the way to check for this bias is to flip all the images and run the test again. If there really is a handedness to spiral galaxies, then we should see the left-handed galaxies outnumber the right-handed ones. What actually happens is the results are unchanged. You again get about 52% right-handed and 48% left handed.

In the words of Pogo, “We have met the enemy, and he is us.” The bias is real, but it’s due to human observers rather than a physical bias in spiral galaxies. This is why you need to be careful with crowdsourced data, particularly when that data depends upon a human judgement call.

For example, a couple years ago a paper came out showing an interesting pattern in the directions of spiral galaxies. Looking in one direction of the universe, there was a slight bias of right-handed spiral galaxies, while looking in the opposite direction there was a slight bias of left-handed galaxies. This is interesting because it is exactly what you would expect to see if there is an innate handedness to the universe.

An apparent bias in spiral galaxies. Credit: Michael Longo

To understand why handedness should vary with direction, imagine you and a friend are standing on opposite sides of a street. You see a car drive slowly along the road. From your vantage point, you see the car move from right to left, and you see the tires clearly rotating in a right-handed (counterclockwise) direction. Your friend, however, sees the car move from left to right, and sees the tires rotate in a left-handed (clockwise) direction. The tires are all rotating in the same way, but you and your friend see the tires rotate with different handedness because you view things from opposite directions. The same is true for the universe. If all the spiral galaxies rotated the same way, we should expect to see right-handed spirals in one direction of the sky, and left-handed spirals in the opposite direction.

This effect is small, but the odds of it being purely chance is about 1 in 10,000. But is this a real signal, or is it noise. The author did account for left-right bias by flipping images and such, but given that human bias can give a 1 in 400 trillion shift, it’s hard to be sure that human bias has been completely accounted for. So it’s an interesting result, but not really a compelling result.

I should emphasize that human bias does not mean that all crowd-sourced data is worthless, or even that this particular work with spiral galaxies is bad. It simply means that crowd analyzed data must be dealt with carefully so that human bias doesn’t creep into your results. But that’s an issue even professionals have to deal with.

After all, everyone tends to favor their gripping hand.

Paper: Michael J. Longo. Detection of a Dipole in the Handedness of Spiral Galaxies with Redshifts z ~ 0.04. arXiv:1104.2815 [astro-ph.CO] (2011)

The post The Gripping Hand appeared first on One Universe at a Time.

The figure below shows the positions of more than a thousand galaxies in the universe.  You might think that tells us things about the history and evolution of the universe, and you’d be right.  But it also tells us something about how we observe the universe.  Knowing the latter is important, because all measurements have biases.  If you don’t account for observation biases, you might mis-interpret your data.

For example, you’ll notice there aren’t many galaxies along the horizontal center line.  This doesn’t represent a huge “void” in the universe, but rather is a clear example of observation bias.  These galaxies are plotted in galactic coordinates, a coordinate system where the “equator” lies in the plane of our galaxy.  This is why you don’t see many galaxies near the center line of this figure.  The galaxies were observed in the visible spectrum, and the gas and dust of the Milky Way absorbs visible light, therefore making galaxies along the galactic equator hard to observe.

It’s also important to recognize that biased data is not worthless data.  Knowing the method of observation tells us we can trust the observed distribution of galaxies in the regions away from the equatorial plane, but not near the equatorial plane.  There are other biases in this graph as well.  For example, this is only a plot of visibly bright galaxies for which we also have redshift data, so there is a “proximity” bias where more of the galaxies are nearby rather than distant.  There are also “bands” of observed galaxies due to the fact that telescopes track through the sky, and it is easier to observe galaxies along a track rather than randomly positioning your telescope.  Of course we can account for those biases as well.

Good science is not just gathering data, but understanding how that data is gathered.  We all have a biased view.  The trick is to recognize our biases and deal with them accordingly.

The post A Biased View appeared first on One Universe at a Time.