Thursday, September 8, 2022

Biases in the Interpretation of Weather and Climate Observations

Placing this week's heat wave into context raises all sorts of questions concerning the veracity of weather observations, including extremes and long-term trends.  The relatively high maximum temperatures at the Salt Lake City International Airport have really cast a spotlight on this.  In fact, yesterday the National Weather Service did some measurements in the surrounding area.  They posted a complete summary thread on twitter that you can access beginning with the tweet below.


The long and short of it is that the area south of the airport is a very hot place.  They found good correspondence between multiple measurements in a dry grass field to the south and the observing site.  Kudos to them for doing this.  There may be few if any meteorologists at the forecast office there who are responsible for the current site and they are doing the best they can to ensure that it records reliable observations.  

Now let's move to a related point concerning biases in the interpretation of weather and climate observations.  Such observations are used for all sorts of applications, from severe weather warnings to long-term climate trends.  I will posit here that there are three kinds of biases in the interpretation of such observations:

1. Instrumentation bias.
2. Representativeness bias.
3. Human bias.

Sometimes #3 is a bigger problem than the first two.  

Instrumentation bias is due to the instrument itself, or the recording and processing processes.  This is especially important for extremes.  The sampling frequency and averaging intervals, for example, can affect the recorded maximum.  I believe NWS maxima are based on a 5-minute average, although I'm not sure what the sampling interval is.  Other instruments may do things differently.  There can be other challenges related to the ventilation of the instrument (radiation shielding and ventilation are really critical for measuring daytime temperatures).  For example, some weather stations in use in the Wasatch Mountains by ski areas are aspirated by the wind and are poorly ventilated when the winds are light, leading to erroneously high temperatures.  

Even if you have an instrument that measures temperature (or some other variable like wind) perfectly, it is essentially a point measurement.  We live in a complicated world, and there can be a lot of spatial variability.  Representativeness bias concerns how representative the observation is of the surrounding area.  The airport, for example, is a hot spot.  During this heat wave, it has reported maximum temperatures 2-4˚F higher than the surrounding stations.  This could be partly related to instrumentation bias, but for this discussion, lets just assume that's a representativeness bias.  On the opposite end are observations collected near the golf courses, which have been considerably cooler than surrounding stations.  The area over which a station might be representative can vary a lot depending on factors like the land-surface or terrain variability.  A wind observation at the mouth of Parley's Canyon, for example, is representative of a very small area.  You move a kilometer to the north or south and you are in an entirely different wind regime.  

Finally we have human bias.  As I said, this can be a bigger problem than the first two.  For weather forecasting, meteorologists must learn to sip from a firehose of information.  This involves making a lot of mental shortcuts.  This can lead to heuristic biases that can sometimes lead to forecast errors.  Similarly, interpretation of climate trends and contexts can affected by such biases.

Human bias in this instance is a compound bias comprised of many biases.  I will steal liberally from Ian McCammon's FACETS acronym for heuristic traps in recreational avalanche accidents.  FACETS summarizes six key heuristic traps that affect backcountry decision making: Familiarity, Acceptance, Consistency, Expert Halo, Tracks/Scarcity, and Social Facilitation.  

Familiarity: In backcountry skiing, parties tend to make riskier choices in familiar terrain. Meteorologists, however, may develop familiarity with observing sites, and this affects their interpretation of data from those sites.  It might, for example, cause them to dismiss an unusual observation because it doesn't fit with their prior experience.  I have done this many times.

Acceptance: Many meteorologists want to be accepted by colleagues as much as backcountry skiers want to be accepted as members of their parties!  If everyone else thinks that an observation or interpretation is correct, you may think so too.  There can also be an anti-acceptance bias in which the meteorologist likes to be a contrarian.  

Consistency: In backcountry travel, this is commitment to an objective.  In meteorology, it might be commitment to a prior forecast or a hypothesis.  When you have been pushing a forecast or arguing for a hypothesis for a long time, it is really hard to change course.

Expert Halo: In backcountry skiing, novices sometimes follow leaders into dangerous situations as they acquiesce to the decision making of a more experienced leader (even guides are not infallible).  Similarly, there is a tendency to give greater credence to more established scientists or forecasters. 

Tracks/Scarcity: This is a tough one to translate to meteorology.  In the backcountry, people may take on more risk if they are competing with others for tracks.  Tracks might not be the right word, but in this latest heat wave, I have seen examples of disinformation due to people rushing to announce records that had not been validated.  

Social Facilitation: It's hard to go against the group, in the backcountry or in meteorology.  On the other hand, sometimes the group is right. 

Scientists are humans too and our biases are real.  On the other hand, the scientific method and processes are designed to minimize these biases and slowly but surely approach a better result or theory.  Feel free to add some additional perspectives. 

5 comments:

  1. My initial response was skepticism when I saw that yesterday hit 107 F. Good on them for going out there with several sensors to verify.

    Do you know if this sensor is aspirated?

    One other thing I remain curious about is the coarse precision of the 5-minute KSLC temperature data available on MesoWest. It looks... rounded, but in some way that only produces values of 102.0, 102.9, 104.0, 105.1, 106.0, 107.1, and nothing in between. Any idea what's going on there? I'd assume they're reducing the precision to the accuracy of the sensor, but I'm not sure why they wouldn't go with whole numbers.

    ReplyDelete
  2. ASOS sensors are aspirated.

    MesoWest receives weather observations from three differing feeds. The order of receipt varies. I think the data resolution does as well.

    There is at least one feed that is to the nearest whole degree. I have seen for example, 105.8, which is 41C rounded off. Why 105.1 I am not sure.

    Ideally, data would be transmitted to the nearest tenth of a degC, but it isn't.

    Jim

    ReplyDelete
    Replies
    1. It appears that at least at the KSLC site, what happens is that the data gets initially recorded or rounded to the nearest whole degF and then converted to Celsius rounded to the nearest tenth. So if it’s 105F (40.556C) out, that gets transmitted as 40.6C, which when you look at Mesowest on your own gets converted directly back to F and then rounded again (in this case, 105.08F which rounds up to 105.1F).

      Same deal with 103F -> 39.44C -> 39.4C -> 102.92F -> 102.9F.

      Delete
  3. The point about the rounding and unit conversion F to C is an excellent example of the need to look behind all data.
    However you slice it and dice it, it was darn hot last week. Hotter than previously reported at this station.
    Did all other stations set records, albeit lower ones?

    ReplyDelete
    Replies
    1. Many sites set records. Impressive event all around.

      Delete