Ensembles of satellite SST

Last week was the previously advertised user consultation meeting on uncertainty information in the SST climate data record (with some sharing of experience with other variables). It was a lively meeting, and my thanks to all participants.

There will be a meeting report, which is in preparation, and outcomes will be assimilated into an update of the SST CCI user requirements document.

An interesting outcome was the appetite among some users for ensembles of sea surface temperature (SST) datasets. This might seem surprising, since satellite-derived datasets are already large, and the prospect of a meaningful ensemble (100 to 1000 realisations was mentioned) would seem to be an obstacle. However, a number of counter-balancing points were raised:

• creating an ensemble can be the only practical way to represent some correlated error structures (particularly errors that evolve on the long term through a multi-decadal dataset, such as those associated with imperfect harmonisation across series of sensors)
• other forms of more detailed uncertainty information, including error covariance matrices, can also represent and large additional data volume
• once a user has an analysis or model run set up for a particular experiment using an SST dataset, it can be much easier for them to explore uncertainty by re-running the same analysis for variants of the SST dataset, than to analyse the dataset uncertainty information and assess its impact (which involves more thinking and work!
• the ensemble generation can involve runs that explore structural uncertainty (effects of data processing choices where there is some element of judgement about parameters, etc) -- and there is no other way to obtain the resulting effects (these choices/parameters generally can't be analytically propagated, and sensitivity to them has to be found by tweaking the choices and re-running)

We can't feasibly produce a full ensemble version of our climate data records within SST CCI phase 2, but we will stay alert to opportunities to progress towards that capability as we are doing the project.

OSTIA water/land mask and new inland water bodies dataset

Laura Carrea and I have been looking at the OSTIA water/land mask in comparison with the new higher resolution dataset from the Landcover CCI project. The OSTIA mask is at 0.05 deg lat-lon resolution, and is used for the SST CCI Analysis product (gap filled daily SSTs); its precise heritage (since being created several years ago) hasn't been re-traceable, so it is useful to check its nature.

The plot below shows for a small area (Baltic Sea) the prevalence of water (according to the LC CCI product) within each 0.05 deg cell which is labelled in OSTIA as land. Red colours indicate the presence of a small fraction of water in an OSTIA "land" cell, and rivers and the many lakes on land are obvious. Dark blues indicate total or near-total water in an OSTIA "land" cell. Some of these are inland lakes no resolved in OSTIA, which is no surprise. There are a few cells around the Baltic coasts that are apparently water-filled (dark blue) yet are "land" cells in the OSTIA mask. Globally, however, such cases are exceptions. In general the coasts don't have a fringe, which suggests that the OSTIA mask is designed such that a cell tends to be labelled "water" if there is any significant fraction of sea within the cell. (If the design were such that a cell was labelled "water" if it contains >50% of sea, then there would be a fringe of intermediate values of %water-in-land-cell all coastlines.) So OSTIA must use a "fat" water mask, rather than a mask that is neutral with respect to land and water.

This next plot shows the prevalence of land in cells labelled water in OSTIA. Here, blue colours indicate a small amount of land, and red a lot of land in the cell. The fact that the whole coastline tends to be fringed with colour confirms that the OSTIA mask is "fat" with respect to water. (That is, truly mixed cells tend to be labelled as sea, so there is a fringe of land-in-water-cell cases along coasts.) However, it is also clear that there are many "water" cells that are in LC CCI completely land (dark red in this picture). These appear preferentially on northern coasts. This is consistent globally, and not just around the UK. This suggests that there is an offset in the N-S direction, roughly half an OSTIA cell in size, in the OSTIA mask relative to the Landcover CCI data set.

Where an OSTIA cell is flagged as sea and is in fact filled with land, there will be (or should be) no satellite SST retrievals ever available for that cell -- the data will always be provided by the gap-filling procedures associated with creating the L4 SST CCI analysis.

How to represent different SSTs in the products

As previously discussed we aim in future products to include not only the skin SST (the primary geophysical retrieval) and, as before, a 20 cm estimate at a fixed local time (to minimise aliasing of the diurnal cycle in the long term trends), but also a UTC-day mean estimate. This combination hits a good fraction of the diverse range of user requirements we collected for depths and time.

We also need to provide an adjustment to the most consistent possible retrieval (default in product is the best available type of retrieval, but one might also want to analyse one type of retrieval through the whole record).

To deliver this information in a GHRSST-compatible form requires some thought, since the retrieval and the various model-derived adjustments all have up to three components of uncertainty as well as their values.

Project team discussions have concluded on the following: to store the best available skin SST as the primary variable, and give a set of adjustments that can be added to this, each with N (1 <= N <= 3) uncertainty components. This is a much smaller data volume than adding all the different SSTs each with three uncertainty components. Data volume is a concern for a significant set of users.

For the convenience of users faced with the complexity of adding adjustments to the primary data, we will need to provide a reader programme that configures the calculation of a desired SST type and its uncertainty. Even nicer would be configuration of the desired fields on the fly on download -- that is a technological solution we aim to discuss with those who will do the CCI programme data portal (invitation to tender currently published).

Uncertainty information in Climate Data Records

We are well into our planning for a User Consultation workshop on representing uncertainty information in our SST CCI products in the most useful way. (Registration is still open.)

Venue: Met Office Hadley Centre, Exeter, UK
Date: 18-20th November 2014.

This workshop will be a two-way discussion between data providers and users. We aim to create a common understanding of: where uncertainty comes from (in this case, uncertainty in satellite sea surface temperature); how to talk about uncertainty unambiguously; how well the uncertainty information that is provided addresses users´ needs; and how to practically use such uncertainty information. It will achieve this through a mixture of oral and poster presentations, activities and group discussions.

We as data producers need to provide uncertainty information that users have confidence in, and have confidence in using. That is, they need to know it is realistic information, and what they can validly do with the information. Achieving this definitely involves increased mutual understanding, so it should be a very stimulating and lively meeting.

Learning Python

After years of relying on IDL for interacting with data, I am taking the plunge and switching to Python. The tipping point was deciding that iPython notebooks are a good way of maintaining the links between results, figures and the code used to generate them. My first plot is based on SST CCI data, of course!

Geoscience Data Journal paper

An open-access journal article describing the SST CCI phase 1 datasets was published today.

It is published in Geoscience Data Journal. I think the advent of 'data journals' over the past few years is a good development. The traditional recourse of trying to shoe-horn a detailed data description into a paper with science results was not ideal, particularly for large complex datasets such as those created by reprocessing EO archives for climate.

The new paper is:

Merchant, C. J., Embury, O., Roberts-Jones, J., Fiedler, E., Bulgin, C. E., Corlett, G. K., Good, S., McLaren, A., Rayner, N., Morak-Bozzo, S. and Donlon, C. (2014), Sea surface temperature datasets for climate applications from Phase 1 of the European Space Agency Climate Change Initiative (SST CCI). Geoscience Data Journal. doi: 10.1002/gdj3.20

1. European Space Agency, ESRIN/Contract No. 4000101570/10/I-AM ‘Phase 1 of the ESA Climate Change Initiative SST_cci’

What differences to use in validation?

Validation is the comparison of (in this case) our satellite SSTs with temperature measured in situ, from buoys, ships, etc. Validation gives assurance that the satellite SSTs are, in a general sense, accurate. However, the comparison is complicated by the fact that different SSTs are genuinely different (geophysical differences), so that the difference between any two data points is a mix of error contributions and true differences. In addition, the SST CCI products include a number of SST estimates, each of which requires validation.

We therefore need to be clear about which in-situ/satellite comparisons will be made, and why. This post records the results of a review of our options, following discussions between myself and Gary Corlett (Leicester).

"Raw differences"

Here, we will compare the skin SST from the satellite to the nearest-in-time depth SST of the in situ measurement. In this case we expect certain systematic differences. (1) There is a geophysical difference based on the ocean thermal skin effect, which is typically of order -0.2 K, but also has a wind-speed dependence which should be clear in night-time differences. (2) There should be a trend in the raw difference with respect to the time separation of in situ and satellite: for example, in mid morning the ocean is typically warming, so in situ measurements after the satellite time will tend to be warmer. However, with respect to things that might affect the satellite retrieval adversely (but not directly the skin-depth SST difference) the systematic dependencies should be small; for example, a systematic effect in the raw differences with latitude should be no larger than we might be able to account for by the fact that mean wind speed (and therefore skin effect) varies with latitude.

"Skin-skin differences"

The idea here is to estimate the skin and depth effects at the time of the satellite observation and add these to the in situ observation. The in situ history is first interpolated to the satellite observation time, so giving an estimate of SST-20cm at the location and time of a satellite SST. Any 20-cm-to-subskin stratification is estimated using a model (usually small, and only ever large for day-time cases) and the skin effect is also estimated using a skin-effect model. There should ideally be no systematic effects with respect to latitude, wind, satellite-buoy-time-difference, etc, because the models are meant to account correctly (on average) for all the geophysical differences (other than those from comparing a point to a pixel, which are assumed to add zero-mean noise). This measure therefore tests the combination of "retrieval + adjustment for skin effect + adjustment in depth".

"Depth-depth differences"

In the SST products there is an adjustment provided that can be added to the fundamental satellite SST retrieval (of skin SST at the time of the satellite) to give an estimate of SST at typical drifting buoy depth (~20 cm) at standard local times of day (10.30 h or 22.30 h). To explore this, the satellite SST-20cm estimate for a standard local time will be differenced with the spatially-matched in situ SST history interpolated in time to the same local time. There should ideally be no systematic effects with respect to latitude, wind, satellite-buoy-time-difference, etc, because the adjustment is meant to account correctly (on average) for all the geophysical effects (other than those from comparing a point to a pixel, which are assumed to be zero-mean noise). This measure therefore tests the combination of "retrieval + adjustment for skin effect + adjustment in depth + adjustment in time". Compared to the skin-skin difference, this tests in addition the time-adjustment of the SST for the diurnal cycle.

"Daily mean depth-depth differences"

Although not generated in existing datasets, there has been a requirement which we are considering to estimate an adjustment to be added to the skin SST from the satellite which would give an estimate of the daily-mean SST at the location of the satellite observation. The day over which the mean is to be estimated is the UTC (i.e., GMT, not local) day which includes the time of the satellite observation. The in situ data would therefore consist of the average of the history of the in situ measurements over a 24 hour period. This comparison therefore tests "retrieval + adjustment for skin effect + adjustment in depth + adjustment to daily mean", and the spread of the results will include the uncertainty effect of estimation of the daily mean SST from a single observation. Systematic effects in the differences should be small.

"System Maturity" CORE-CLIMAX style

The SST CCI project is to create both new SST data and to prototype a system for how this SST climate data record can continue to be routinely provided in the future. The "system" is a processing chain that takes inputs (satellite radiance data, auxiliary data etc) and transforms these into SST products. It consists of something like 100000 lines of code, installed at the facility for Climate and Environmental Modelling from Space (CEMS) in Harwell.

CORE-CLIMAX [no kidding] is a European project. Within its scope is development of a means of encapsulating how "mature" systems for delivering climate data records are. It is quite instructive for a team like ours to evaluate itself against the various criteria in the "System Maturity Matrix" they propose. We just did a self evaluation, for AVHRR and analysis products, and find that in its current state, the project straddles a "research capability" and an "initial operations capability" in most areas. That seems right -- it is exactly where we would expect to be at this stage, working on science and also towards a functioning, sustainable system.

Here are our self assessment results. The green shaded boxes show the range of "maturity" of different aspects of the project within each of the metric categories (software readiness, metadata, user documentation, uncertainty characterisation, feedback/access and usage).

Heat Content vs Surface Temperature

The Guardian have (after inputs from ESA) updated the article discussed here to say that the "lousy indicator of climate change" comment applied to surface air temperature rather than sea surface temperature. Well, I suppose that is slightly better from my point of view, but I still don't agree with the point!

"Global surface temperature" time series are generally made up of surface air temperature over land and sea surface temperature over the oceans. Together, surface temperature, precipitation, wind and solar radiation are the principal elements that define our experience of weather and climate. These matter to people, and we need to present any climate changes (anthropogenic and natural) in terms of these elements. Surface temperature is not a simple indicator of the accumulated heat in the climate system, that is true. But it is an indicator of great relevance to society and the environment.

Did ESA say our data are "lousy"? [Answer: No]

Talking last Friday at the Royal Society (London) about ESA's Climate Change Initiative programme was an interesting experience. I gave a presentation on the title of "Ocean Warming". The idea of the talk was to argue that signals in data from the CCI teams are consistent with an "oceanic heat burial" hypothesis published in March this year.  This is the idea that surface temperatures during the 2000s have been fairly level (see Figure 1 below), but at the same time, the ocean overall has continued to gain heat because of greenhouse gas forcing of climate. This apparent contradiction is resolved by changes in tropical circulation that have 'hidden' heat below the ocean surface over the past decade. Patterns consistent with the associated circulation changes can be seen in sea surface temperature, ocean colour and sea level data from CCI.

Figure 1. The measurements from space (red line) are SST CCI data, which show the same year-to-year picture of sea surface temperature changes as the in situ only data from the Hadley Centre in blue. Within the next 3 years of the project, we aim to extend this time series at both ends.

There were other talks, too, including a very positive introduction by the Rt Hon David Willetts (Minister for Universities and Science), one on the cryosphere as seen in CCI data by Andrew Shepherd, and talks by representatives of ESA. 

So, it is interesting what the press picked up. The Guardian have published an article following the meeting with the headline Apparent pause in global warming blamed on 'lousy' data. Within the article it says:

Now, Stephen Briggs from the European Space Agency's Directorate of Earth Observation says that sea surface temperature data is the worst indicator of global climate that can be used, describing it as "lousy".

If you read it quickly, you might think that ESA meant our SST data are "lousy"!

In fact, the point being made was that the energy required to warm just the ocean surface and the surface air temperature is a tiny part of the total energy that the Earth is gaining because of greenhouse gas forcing of climate (see Figure 2). Turning it round the other way, this means that variations in the rate of surface temperature change do not necessarily imply that the Earth has stopped gaining heat. The heat can still be going into other, much more dominant, components of the climate. This is scientifically correct, and indeed, my talk showed a specific example of that. (There will inevitably be year-to-year, decade-to-decade variability in surface temperatures -- weather doesn't stop because of global warming.) 

So, Stephen Briggs' words did not mean he thought our SST data were terrible, despite the impression given by the headline!

In my view, there are many compelling reasons to use surface temperature as an indicator to describe global climate. People experience temperature directly (albeit, subjectively), it is relevant to human comfort and health, temperature (with wind) drives evaporation, it is relevant to agriculture (on land) and fisheries (at sea), we have instrumental records of temperature going back over one hundred and fifty years, etc.

In contrast, the total energy gain in the climate system would seem rather remote to most people, I expect -- although it would be great if everyone understood physics and climate science sufficiently well to grasp its significance.

Figure 2. Analysis of heat content of climate system, from IPCC AR5 WG1 Ch3 Box 3.1.

New version of SST CCI ATSR dataset -- v1.1

A bug-fixing upgrade to the SST CCI ATSR data is now available from the data centre. Relative to v1.0, the newly released v1.1:

• Completes the record up to the end of the AATSR mission (April 2012, cf. Dec 2010 for v1.0)
• Fixes persistent holes from missing data after midnight
• Restores missing files
• Fixes various metadata bugs

The DOI of the new version is 10.5285/79229cee-71ab-48b6-b7d6-2fceccead938.

Give us local expert feedback!

The annual GHRSST science team meeting has just finished. The meeting was held in Cape Town, and it was great to learn about the interesting ocean dynamics around the coasts of southern Africa and the practical uses to which satellite SST datasets are put. Mostly in the SST CCI project we focus on global scale assessment of our products, but this meeting was useful in raising the issue of long-term variability and change in the datasets at small scales in coastal regions. For example, in the image below, upwelling regions off the western cape are resolved in the SST CCI analysis dataset  (narrow stretch of colder water running north along the coast from Cape Town). The fidelity of the variability in this feature from weeks to decades has never been assessed by us specifically: although regional applications of SST CCI data were tested in the Climate Assessment Report, none of the trail-blazer users were focussed on this oceanographically interesting area. Changes in SST in this region have practical implications for fisheries and coastal industry. Users of our data please note: if you work with our data and get insight into its utility or limitations in specific areas like this, we would like to hear from you, it is useful feedback!

System Requirements at CCI Programme Level

The Climate Change Initiative programme as a whole has defined System Requirements and Data Standards in two documents. I, Owen, SST CCI project manager (Hugh) and the SST CCI system engineering team (Ralf and Martin) reviewed those documents against our plans last week. This was partly in response to a query from ESA, and we thought it worthwhile to look at the issues in detail for our own benefit from the point of view of making sure the system we are building is as good as possible.

The brief conclusions are:

1. SST CCI is fully in line with the data standards, since the data standards are compatible with GHRSST standards
2. The SST CCI project will internally meet 50% of the CCI programme system requirements (SRs) in Phase 2
3. We think 25% of the SRs can only be done through programme-level investment and activity (some of which is foreseen, e.g., data portal work will be commissioned by ESA)
4. We think 12% of the SRs are relevant only in the context of sustained operations after Phase 2, and will require additional work on the SST CCI system at that time
5. We question, disagree with or don't understand the remaining requirements

It was useful to us to have identified through this exercise the SRs that fall into category 4.

Petrenko et al (2014)

"Evaluation and selection of SST regression algorithms for JPSS VIIRS" by Petrenko et al has been published in the Journal of Geophysical Research: DOI: 10.1002/2013JD020637. It compares validation statistics for a number of coefficient-based algorithms for sea surface temperature (SST) on a common basis, which is a good thing to see. The formulation from the Ocean and Sea Ice Satellite Application Facility turns out to be the best performing.

A point of interest is that the OSI-SAF algorithm was not the one that gave the minimum standard deviation against drifting buoys. An algorithm from the Navy Research Laboratory gave considerably lower spread (0.36 K compared to 0.42 K). However, the NRL algorithm was rejected on the basis that its sensitivity to SST* was greatly suppressed -- only 40%. This would mean, for example, that diurnal variability would be under-estimated by 60% using the NRL algorithm. This trade-off (apparent "accuracy" vs. sensitivity) has been discussed in the literature before, but as far as I recall, this is the first paper not involving SST CCI team members that has seriously used sensitivity as a criterion in algorithm selection.

*Sensitivity is the amount the satellite SST changes per unit of real SST change. Ideally, we would want 100% -- i.e., 1 K change in the satellite SST for a 1 K change in the real SST. However, this does not happen where a retrieval relies heavily on information brought to the observation ("prior information").

Collaborative agreement with JAXA

Within their 5th Research Announcement of opportunity for the Global Change Observation Mission, JAXA have accepted an SST CCI proposal "SST Algorithms for AMSR-2 Intended for Use in long-term Climate Data Records". We look forward to this collaboration.

The collaborative agreement is being finalised. The objectives are:

1.     To carry out development and testing of optimal estimation to generate SSTs from passive microwave satellite instruments using optimal estimation retrievals.
2. To ensure that the optimal estimation of SSTs from the microwave measurements is done in a way consistent with infrared methods.

Nadir - Forward Shifts in ATSR Multi-mission Level 1b archive v2.1

This summarises some results from Owen Embury looking at whether, with the latest ATSR multi-mission archive, there is still a need (as in Phase 1) to shift the forward view to better line up with the nadir view for the ATSRs. We always knew that our empirical shifts in Phase 1 were only approximate and hoped that the v2.1 archive would have resolved the issue completely. Bottom line: it is an improvement, but not perfect.

Here is a scene with the v2.0 nadir-forward collocation. Note that in the Nad-Fwd differences there are red and blue (warm and cold) differences along SST fronts (wavy lines) in the thermal (middle panel) and the island have a bright edge and dark shadow. Both these effects imply the two views are not perfectly collocated.

Here is the equivalent from v2.1. The effects are much less, but the opposite effects can just be discerned.

Here is the same scene with a shift of a pixel. The fronts are less obvious and the shadowing of the land is not really much different. For SST this is suggests an improvement.

Owen did a further analysis, minimising nadir-forward variances across many small scene extracts, to see if the offsetting effect is constant in time. It appears not to be, as per the example below:

At the beginning of the AATSR mission, the best shift of the forward view is -1 pixel along track and 0 across track on the left side of the swath -- but from 2007, it appears better to shift across track by +1. For other sensors, the equivalent curves suggest the instrument geometries are even less stable.

However, the good news is that pretty much all of the required shifts are no greater than 1 pixel, which is an improvement over v2.0, where shifts up to 3 pixels were required.

Datasets from SST CCI and their DOIs

The datasets generated to date by ESA's Climate Change Initiative project for Sea Surface Temperature  are available from the Centre for Environmental Data Archival via the page www.neodc.rl.ac.uk. Possibly it is most useful to go to the dataset pages using the DOIs of the datasets:

For SSTs derived only from the Advanced Very High Resolution Radiometers:
http://dx.doi.org/10.5285/782305d4-1ded-43f9-a0ed-40cb41ff0a43

For SSTs derived only from the Along-Track Scanning Radiometers:

http://dx.doi.org/10.5285/fe7a9d81-cfc0-4023-90da-0be37b803bc7
(This points to v1.0.  V1.1, addressing a few problems in v1.0, will be coming along soon.)

For a gap-filled, daily blend ("SST CCI analysis") of the above datasets (with good feature resolution):

http://dx.doi.org/10.5285/878bef44-d32a-40cd-a02d-49b6286f0ea4

To get data, you need to register just your e-mail address with the data centre. If you try to access data without logging in, you get a page that says "access to the dataset is restricted". This is misleading: there is no restriction on obtaining SST CCI data, you simply need to be logged in to the data centre.

Below is an example day from the SST CCI analysis. This image represents the sea surface temperature (SST) across the global oceans on 5th January, 2010. It is made by blending satellite-derived estimates of SST and filling gaps by an optimal interpolation -- a process referred to as 'analysis'. This SST analysis was created using data from the Along-Track Scanning Radiometers (ATSR) and Advanced Very High Resolution Radiometers (AVHRR), analysed using the Met Office OSTIA system. 

The particular value of the SST CCI dataset is that it has relatively high feature resolution, refers to a well-defined type of SST (temperature at 20 cm depth, daily average) and, unlike most satellite SSTs, is independent of in situ observations. The full dataset covers 1991 to 2010 and was created in Phase 1 of ESA's SST Climate Change Initiative (SST CCI). 

What is a sustainable system?

The project has two aims. The most obvious is to produce "climate data records" for sea surface temperature. SST is an "essential climate variable", which means that to understand and track climate variability and change, we need high quality records of how SST changes over time. With sufficient remote-sensing know-how, this can be achieved using infra-red imagery from Earth observing satellites.

The project has a limited lifetime, but the need for climate data records won't disappear. Therefore we have a second aim: in the process of delivering data, we will build a software system that can then be sustained to provide data in future. We don't know how that will be funded, at present, but nonetheless over the next 3 years the project will develop its software to a stage of being, in the jargon, "pre-operational".

 The system won't be able simply to be run untended, even in an sustained (or "operational") mode, without maintenance of the science, and cyclic improvement to our techniques. This is because satellites come and go, and to get new observations to the standard required for inclusion in the SST CDR requirements ongoing Earth observation development work. Moreover, overtime, we discover how to make CDRs better, and therefore should reprocess the whole dataset consistently in the improved manner.

Thus, the system includes software and human experts, in a cycle of sustained production of data (as the observations come in) and periodic reprocessing. Conceptually it is like this:
 

What is this blog about?

This blog relates to a three year project to create climate data records of sea surface temperature, based on observations of Earth from satellites in space. The project is sponsored by the European Space Agency (ESA) and will run from January 2014 to March 2017.

I am the "science leader" of the project, and want to use this blog for a number of purposes.

The blog will be a useful aide-memoire for me of the discussions and conclusions that take place in the project that don't arise within formal, minuted meetings. We could use a private wiki for this, but using a blog achieves a second purpose: exposing to the public an example of how science is done. And how is science being done in this case? Collaboratively. We have a team of 8 institutions in this case, from round Europe. So, lastly, I hope the blog will help with internal communication -- hopefully team members will subscribe and get informed about aspects of the project they were not directly involved in.

We have a formal website: www.esa-sst-cci.org. That says more about the project, team, our outputs (data and reports), which I won't repeat here.

Chris Merchant
University of Reading