Was Preisendorfer's Rule N used in MBH98 tree ring PC calculations?

Mann et al. have recently argued that they can salvage MBH98-type results using correct PC calculations under "the standard selection rule (Preisendorfer's Rule N) used by MBH98". http://www.realclimate.org/index.php?p=8  They say that this method permits them to retain 5 PCs in the North American network. Since the bristlecones are in the PC4, this expanded roster still permits them to imprint the NH temperature reconstruction. We have discussed elsewhere many issues regarding the robustness and statistical significance of this calculation. Here I consider the narrow issue of whether this method was actually "used by MBH98" for tree ring networks. I have been able to closely replicate the diagram published at realclimate.org on Nov. 22, 2004, said to be an example of the selection method used in MBH98. I have tested the 19 network/calculation step combinations used in MBH98 and, in 18 of 19 cases, the selections from the Preisendorfer-type calculation are inconsistent with the reported selections at the Corrigendum SI. In some cases, the results are higher; in some cases, lower. In three calculations, different selections are taken from the same network in different calculation steps - a result inconsistent with the stated policy. We remain puzzled why Mann et al. continue to refuse to provide source code for MBH98 calculations and why climate scientists do not expect them to do so.

Statements in MBH98

First, there is no mention in MBH98 or the MBH98 SI that Preisendorfer's Rule N was used to determine the number of retained PC series for tree ring networks. The only pertinent reference in MBH98 was as follows:

Certain densely sampled regional dendroclimatic data sets have been represented in the network by a smaller number of leading principal components (typically 3–11 depending on the spatial extent and size of the data set). This form of representation ensures a reasonably homogeneous spatial sampling in the multiproxy network (112 indicators back to 1820). [our bolds]

This statement contains no reference to the use of Preisendorfer's Rule N. 

In connection with the calculation of temperature principal component series, a different calculation, MBH98 does refer to the use of Preisendorfer's Rule N as follows:

a conventional Principal Component Analysis (PCA) is performed... An objective criterion was used to determine the particular set of eigenvectors which should be used in the calibration as follows. Preisendorfer’s selection rule ‘rule N’ was applied to the multiproxy network to determine the approximate number N_eofs of significant independent climate patterns that are resolved by the network, taking into account the spatial correlation within the multiproxy data set.

Before trying to interpret these two statements from a text analytic point of view, I will make four quick points about rules for deciding the number of PCs to retain:

• The briefest survey of PC literature will show that there are many approaches to selecting the number of PC series to retain and Preisendorfer's Rule N is far from being a "standard selection rule".  

• in fact, Urban, in a presentation about PCs cited on Jan. 6, 2005 by Mann at realclimate stated that the choice was subjective as follows: 

  It should be noted that because the goal of PCA is essentially utilitarian, the choice of how many axes to retain is ultimately subjective. In practice, either 2 or 3 axes are retained, simply because it is difficult to project more than this onto a printed page.

• Overland and Preisendorfer [1982] themselves argued that being significant under Rule N was only necessary for significance; they did not argue that it was sufficient.  

• The real test for retaining a PC series is not whether it is significant under Preisendorfer's Rule N (or some other such rule), but whether it is scientifically significant,. For example, Franklin et al. [1995] stated:

In the final analysis, the retained components must make good scientific sense (Frane & Hill 1976; Legendre & Legendre 1983; Pielou 1984; Zwick & Velicer 1986; Ludwig & Reynolds 1988; Palmer 1993).

Now, from a text analytic perspective, a reasonable reader might conclude that the difference in description of the PC retention policy in the two cases - tree rings and temperatures - pointed to the use of different procedures in the two calculations. In fact, the form of PC calculation in the two calculations differed: we have determined that the temperature PC calculations were centered calculations, while, as we've pointed out in our recent articles (and earlier), the tree ring PC calculations were not conventional centered calculations. Mann et al. have recently (Jan. 6, 2005) acknowledged that they did not use a "standard centered method" so their use of an uncentered method is no longer in dispute.

Applying Preisendorfer's Rule N to Tree Ring Networks 

The real test for whether Preisendorfer's Rule N was used in MBH98 was whether the actual number of selected PCs can be replicated using this method.

The actual retentions for each calculation step/network combination were not provided in MBH98, its SI or at Mann's FTP site. The first complete listing of actual retentions came in the Corrigendum SI (July 2004). Even the Corrigendum SI contains no summarized listing: the following table was collated from the Corrigendum SI and shows the number of retained PCs by calculation step-network combination. (It was impossible to deduce this table with the additional disinformation of Mann et al. [2003] that 159 distinct series were used, since only 139 distinct series were actually used. Any such deduction attempts were further blocked by erroneous listings of the number of series used in the AD1450 step and the erroneous non-use of 6 available series in the AD1500 step. These do not affect early 15th century results, but frustrate attempts at replication.) 

Table 1. Proxy series used in MBH98 (collated from Corrigendum SI, July 2004), showing the number of retained PC series by network-calculation step combination. *: The total number of series used in the AD1450 step is incorrectly stated in MBH98 as 24 (but error is not reported yet). Six series available in the AD1500 network are not used.

The first hints that a Preisendorfer-type policy had supposedly been used in MBH98 came in our Nature correspondence. In response to our observation of the error in their PC methods, Mann et al. had noticed that, under correct PC calculations, the bristlecone pine pattern was demoted from the PC1 to the PC4. They argued that they could still salvage a hockey-stick shaped series using a Preisendorfer-type calculation on the AD1400 North American network. The calculation published on Nov. 22, 2004 at realclimate showing the implementation of a Preisendorfer-type calculation on the AD1400 North American network was originally submitted in our Nature correspondence. We had seen this diagram and calculation in August 2004 and had fully considered it in our GRL submission - in fact, it contributed to the approach taken in our GRL submission, which differs substantially from our previous Nature submission. 

Figure 1 below, http://www.realclimate.org/index.php?p=9 (Nov. 22, 2004), and the two tables are all taken from realclimate, illustrating the application of the supposed Preisendorfer-type calculation. (The original section from Preisendorfer is re-typed here for reference.) The blue and red lines show the simulation results (using AR1 models of the AD1400 North American network) under MBH98 and centered PC calculations respectively; the red and blue points show actual results from the MBH98 and centered PC methods respectively. Preisendorfer's Rule N selects PC series as long as the actual eigenvalue exceeds the simulation. For the MBH98 method, 3 eigenvalues are clearly separated under Rule N and perhaps 7 in a centered calculation. This result is strangely described by Mann et al as follows:

" In the former case, 2 (or perhaps 3) eigenvalues are distinct from the noise eigenvalue continuum. In the latter case, 5 (or perhaps 6) eigenvalues are distinct from the noise eigenvalue continuum.

It seems obvious that the selection of 2 (rather than 3) eigenvalues in MBH98 cannot be directly justified on this diagram without appeal to some still unstated method. 

Figure 2 below shows my replication of the above calculations. The left panel shows the figure from realclimate.org (repeating Figure 1 above), while the right panel shows my emulation, using the script here. The salient features of the methods are obviously captured.  

FIGURE 2. AD1400 North American network - Preisendorfer-type calculations Left panel: Mann et al. [realclimate]. Points - NOAMER netowrk; lines - simulations. Blue - MBH98 decentered; red - centered. Right panel:  Emulation of calculation in left panel.

Testing on Other Network-Calculation Step Combinations

I now test the reported PC retentions for each network/calculation step using this emulation..

Stahle/OK: MBH98 only used one network/directory combination here. Three PCs were retained in MBH98.  Here the third PC is obviously insignificant, but is used anyway. The difference between the MBH98 (blue) and centered (red) results is very little here - presumably because Stahle pre-whitens site chronologies. 

Stahle/SWM: In every example, there is at least one inconsistency with the supposed Preisendorfer policy: a) non-use of the PC2 in the AD1400 network; b) non-use of the PC2 in the AD1450 network; c) non-use of the PC3 in the AD1500 network; d) use of the PC4 in the AD1600 network; e) use of the PC4-PC7 in the AD1700 network; f) use of the PC4-PC9 in the AD1750 calculation step; and g) the increased retention of PCs in the AD1750 calculation step, even though the same AD1700 network is used.

South America: In the South American network, the use of the PC3 in the calculation steps after AD1750  is questionable under the new policy, as is the increment of the 3rd PC in a later calculation step. This network was also the source of major inconsistencies between the listed sites and the sites actually used (see Corrigendum), as the network was reduced from 18 sites to 11 sites. The reasons for the inconsistency provided in the Corrigendum are incorrect, as I'll post on another occasion.

Australia/NZ: The retention of the PC3 in the AD1600-1730 steps and the retention of the PC3 and PC4 in the AD1750+ steps is inconsistent with the new policy. (The non-use of an earlier phase of the network is also problematic.)

Vaganov: The Vaganov network is a network of Russian sites. There is relatively little difference between centered and uncentered methods in this network. This appears to be because of pre-whitening used by Vaganov in developing tree ring chronologies. The pre-whitening dramatically reduces the autocorrelation in the network. These graphs are very important in assessing MBH98 practices, as they show series with large Preisendorfer significance, which are not used: the PC2 in the AD1450 network, the PC3-5 in all steps from AD1600 on. 

North American Network after 1400: Again, there are inconsistencies in the implementation of the supposed policy: a) the non-use of the PC3 and perhaps PC4 in the AD1450 step; b) the use of the PC5-PC6 in the AD1500 step; c) the increase in retained PCs (within the same network) from the AD1450 step to the AD1500 step; d) the non-use of the PC8 in the AD1600 step. 

Summary

The following table summarizes the above replications:

It is obvious that application of the Preisendorfer Rule N method to actual networks does not yield the PC selections archived at the Corrigendum SI. In some cases, more PCs are archived; in other cases, fewer PCS: there is no obvious pattern. In addition, in 3 cases, different selections were made from the same network, e.g. 7 PCs were selected from the AD1700 SWM network in the AD1700 calculation; the same network was used in the AD1750 calculation step, but this time 9 PCs were selected. This would not be permitted without some still unreported adaptation of the method.

Perhaps there is some common factor to the above process that we have not discerned - however, we are confident that no other third party in the world has been able to discern the pattern. Had Mann et al. archived their source code for these calculations (and for other calculations), then these issues would not be a matter of speculation.

Notwithstanding all of the above, as far as I'm concerned, the main issue is whether the PC series so selected are significant in a scientific sense, rather than a data mining sense. We've provided many caveats in our E&E article to reliance on the bristlecone pine series as arbiters of world temperature history - whether they are in a PC1 or a PC4. However, if Mann is to insist at this late stage that the selection of 5 PCs is justified on the present record, it seems evident to me that an explanation is required for exactly how the Preisendorfer-policy set forth here can be reconciled with actual retentions. Perhaps it can, but I've so far been unable to figure out the secret. These guessing games are also pointless. I invite any readers that have got this far should express their objections to the U.S. National Science Foundation and to Nature that this important source code should continue to remain undisclosed. 

Franklin, Scott B., Gibson, David J., Robertson, Philip A.,Pohlmann, John T. and Fralish, James S. (1995), Parallel Analysis: a method for determining significant principal components, Journal of Vegetation Science 6: 99-106.

Overland and Preisendorfer [1982], "A Significance Test for Principal Components Applied to a Cyclone Climatology", Mon. Wea. Rev. 110, 1-4. 

Footnote: Preisendorfer’s Rule N is a simulation method based on white noise, stated as follows:

This rule of PC selection is a dominant-variance rule and is based on a Monte Carlo procedure which simulates sampling from N_p(0,Σ), with  Σ=σ² I_p. The null hypothesis is that our n x p data matrix Z has been drawn from such a population. By following the procedure outlined below, we can systematically accept or reject this hypothesis. Let R be the centered random data set so formed (cf 5.4 and 5.5). Forming S=R^TR for each such sample, we then build up a cumulative distribution for each of the ρ = min(n-1,p)) non-zero eigenvalues λ_j, j = 1,…,ρ. We can then compare the data eigenvalues of the given n x p data set Z, one by one, with these cumulative distributions. The details follow.

Construct (say) 100 independent realization of each of np variates from N(0,1). Form the n x p matrix R as in (5.4) and (5.5). This is the random n x p counterpart R to the given n x p data matrix Z. The ω^th realization R(ω) of the centered R results in an ordered sequence of non-zero eigenvalues:

λ₁(ω) > …> λ_ρ(ω), ω = 1, …,100, ρ = min(n-1,p).

Write

U_j(ω) = λ_j(ω) {  ρ^-1 Σ_k=1:ρ λ_k(ω)   }^–1,  ω=1:100, j=1:ρ

For each j, order these (after relabelling) as

U_j(ω₁) < …U _j (ω₁₀₀)

and set

σ_j(05) = U_j(ω₅) ;  and σ_j(95) = U_j(ω₉₅) ;

These σ_j values define the 5% and 95% points on the cumulative distribution for the jth random eigenvalues.

 For the given data matrix Z with its associated ordered set of non-zero eigenvalues, write:

V_j = d_j {  ρ-1 _Σk=1:ρ d_k }^-1 j= 1,…ρ

Thus we have

                     Rule N: p’ is the greatest j for which V_j > σ_j(95); 0 if no such j exists.