Back in Time...

Many of the products we provide on weather.utah.edu derive from computer models run by the National Centers for Environmental Prediction, or NCEP for short.  Centers is plural because NCEP actually consists of nine National Weather Service centers that deliver national and global products from weather warnings and forecasts to computer model guidance.

We are fortunate in the United States that government weather data from NCEP and other government agencies is for the most part freely available.  This enables groups like mine and others to drive products like those on weather.utah.edu and to develop new forecast techniques.  Many private companies can integrate NCEP model forecasts into their operations, from IBM to Amazon.  This has huge benefits for the nations economy and resilience to weather hazards.  In many other countries, this is not the case as government groups may charge for these products.

When I began my education in the late 1980s, there was essentially no ability to get digital model output from NCEP.  Model forecasts came via difax machine and you only got a few levels.  In my office, I have a small photo of myself standing in front of these difax charts on my graduate day in 1989.  The only way to forecast the weather during this period was to go to the department's map room and look at these charts.  

Around that time, some groups began to explore the possibility of obtaining and using the NCEP model output in novel ways.  One was here at the University of Utah where John Horel (currently our Department Chair), Lloyd Staley, and Tim Barker began to access and display NCEP model forecast data obtained via satellite broadcast.

  

Critical for this effort were the initial products developed by Unidata, a community of education and research institutions supported by the Unidata Program Center in Boulder, CO, which was and is still funded by the National Science Foundation.  This community and the Unidata Program Center exists today and pushes the envelope of what is possible in atmospheric sciences education and research.  

My first foray into really taking advantage of these developing capabilities was in graduate school in 1994 when I began to work with Cliff Mass, Mark Albright, and others at the University of Washington to develop a real-time forecasting system for the Pacific Northwest.  Essential for that effort is the timely availability of model forecasts from NCEP to create the initial analyses and other information needed to run such a forecast system.  

In my 25+ year career at the University of Utah, model data from NCEP has been integral for my teaching efforts and the products we produce on weather.utah.edu.  In the case of the latter, we not only use that model data directly, but we also generate derived products that add considerable value to the raw model output and push the envelope of what is possible from a weather forecast standpoint.  Probably the most popular are the "forecast plumes" that we produce for Alta and many other sites and involve the application of downscaling techniques and snow-to-liquid ratio algorithms to produce higher-resolution forecasts.  The one below is for Alta and suggests we may see a return to powder skiing early next week.  Much thanks to Trevor Alcott and Mike Wessler, former students who developed much of this capability.  

However, every now and then we go through a rough patch and we're going through one now.  There is often a mismatch between the demands of the users of NCEP model products and their ability to provide them. Although some NCEP products are now available on the cloud, some of what we use for our ensemble products is not.  On April 20th, to avoid their servers being overwhelmed, NCEP throttled access on some of their servers, limiting user access to 120 hits per minute.

That sounds like a lot, but to generate the NAEFS products, we need to download 1508 files.  Prior to their April 20th "speed limit," we parallelized the downloading to speed things up.  

When NCEP throttled access, everything broke.  Our code was not built to deal with blocked access.  It took me some experimenting and testing to figure out how not to invoke the wrath of NCEP and how to deal with things if we do.  I've had to basically rewind the clock, going from parallel to serial downloading of the data and also building in delays to slow the code down.  Sadly, this isn't progress and it will probably result in the NAEFS products being ready about 15-20 minutes later than previously, although that's a guess as I never saved runtime information from the parallelized code before NCEP broke it.  

For those of you who use this data, my apologies for the delays.  If I was on the ball and monitoring things, I could have probably seen it coming and perhaps dealt with this in advance, although I confess that the time I have to actually program is pretty limited.  It may be a few days until things are running smoothly again.