Skip to main content

Full text of "ERIC ED110519: A Note on Allocating Items to Subtests in Multiple Matrix Sampling."

See other formats


/DOCUMENT* RESUME' 



ED 110 519* 



TM 00a 794 



AUTHOE 
TITLE 

INSTITUTION 

REPORT NO 
PUB DATE 
NOTE 



Shoemaker/ David H. . - 

& JLo*te-on allocating Items to Subtests in Hultiple 
Matrix Sampling* . 

"Southwest Regional Laboratory for Educational 
P.esearch and Development, Los- Alatfitqs, Calif* 
SWRL-TM-3-72-05 
13 Jun 72 . 
T9p. 



EDBS PRICE 
DESCRIPTORS 



IDENTIFIERS 



• {JF-$0.76 HO$i.58 PLUS POSTAGE < . ( 

*Iteta Sampling; *Matrices; ^Sampling; *Standard Errc: 

"of Beasuremejit; ^Statistical' Analysis; Statistical 
Bias; Testing Problems # * 

Jackknife; *Multi"ple Matrix^Sampling 



•ABSTRACT S f \^ m 

Investigated' -empirically; through .post mortem ^ 
itemrexaainee sampling were t*hS relative merits of two. alternative 
procedures for allocating items to subtests in multiple matrix 
sampling a*nd the 'feasibility of using the jackknife in approximating, 
standard errors. of esjtim.ate. The results indicate clearly that a 
partially balanced incomplete block design is preferable to random 
sampling in allocating items to subtests* The jackknife was found to. 
-better approximate standard: errors of estimate in the "latter item 
allocation procedure than in the former. .These and other results are 
discussed in detail* (Author) - , 



\ 



* Documents acquired by ERIC include many infprmal unpublished * 

* materials not available from .other sources/ ERIC makes every effort * 

* to obtain the best copy available* nevertheless** 5,t* ms of marginal * 

* reproducibility" are' often encountered and this affects the quality * 

* of . the microfiche and hardcopy reproductions ERIC makes available * 

* via the ERIC Document Reproduction Service (EDRS) ♦ EDRS&is not * * 

* responsible for the quality of^the original document.* Reproductions * 

* supplied by EDRS are the best that can # be made from the original* * 

***************************** ********* ********************************** 



x—i 

o 

x — . 




DATE 

vn- 



SWHL 



;P£«Mi$$iON TO 'ft£PflCOUC£ T>-l$ COP*- 
9»GH:ED UATEOVAl HAS 8£EN*GPANT£0 BY 



SOUTHWEST REGIONAL. LABORATORY 
TECHNICAL MEMORANDUM * 



fOEBlC ASO ORGANIZATIONS OPERATING 
WNCEH ACRKUEN^SWITM THE NATlONAt IN- 
STATE OF EDUCATION PURSER PEPRO- 
OUCTiCN OUTSIDE. ThE £«C . SYSTEM RE- 
OWOES PERMtSStON 0^ THE COPYfcGMT 
OVteNE 9 



June t3, 1972 - 
TM 3-72-05 



U5 OEPARTMEMTOF HEALTH. 
* EDUCATION* WELFARE 
' NATIONAL INSTITUTE OF 
EDUCATION 
TrflS DC.-UWENr HAS BEEN REPRO 
OUCSO EXACTLY RECEIVED FROAA 
THE PERSCN OR ORGANIZATION ORIGIN 
ATtNGIT POINTS OF VIEW OR OPINIONS 
STATED DO NOT* NECESSARILY RSPRE 
SENT OFFICIAL NATIONAL INSTITUTE OF 
EDUCATION POSITION OR POLICY 



TITLE: 7L NOTE ON ALLOCATING ITEMS TO SUBTESTS IN MULTIPLE MATRIX 
SAMPLING . . f~ 

AUTHOR: David M» Shoemaker \ 



ABSTRACT 

Investigated empirically through post mortem item*-examinee sampling 

were the relative merits of two alternative procedures for allocating 

items to subtests in multiple matrix s'ampling and the feasibility of 

using the jackknife dn approximating standard errors of estimate. The 

results indicate clearly that a partially balanced incomplete block 
• • • 

design is preferable to random sampling in allocating items to subtests. 
The jackknife was found to better approximate standard errors of estimate 
in the latter item allocation procedure than, in the former. These and 
other results are discussed in detail. ' < 

i 



9 

ERIC 



p.tit docywent n intended to*. *.itcjmal staff distribution and use. Pernlsaion Co reprint or quote fr*?a thi* workirf K 
docuaent. -holly or In part, should be obtained froa SWRL. 4665 LampSOn Ave • , LOS AjamitOS,. 



A NOTE ON ALLOCATING ITEMS TO SUBTESTS IN MULTIPLE MATRIX SAMPLING 

i ? «* 

David .M« Shoemaker 

. * ! " • • 

Multiple matrix sampling or, more popularly:, item-exami it*e sampling, 

» 

is a procedure in which a set of K test items is subdivided randomly 

>into Jt subtests containing k items each with each subtest administered 

to n examinees selected randomly from the population of Re; 4iiinees« 

Although each, examinee receives only a proportion of'fche K test "items, 

the equations given by Hooke (1956) and Lord,, (1960) permit the, researcher 

to estimate parameters of the test score distribution which would have 

been obtained by tes tjlng all N examinees over .all K test items. Because 

numerous combinations of _t, k, and n are feasible in any investigation, 

the researcher. must coi&e to grips with several questions about how the 

procedure should be implemented* "Ifow should items be allocated to 

subtests? 11 , is one important question requiring an answer and is^the one 

addressed specifically herein; concomitantly, the feasibility of using 

the jackknife procedure for approximating standard errors of estimate 

in multiple matrix sampling is considered .in some detail, 

A basic requirement in multip'le matrix sampling is that k items 

from the Kr item 'population are allacated randomly to- each subtest. 

However, in constructing the Jt subtests , four, general item allocation 

* < * - f . ♦ 

procedures are*BPSsible--each of which is described mone appropriately 

as restricted random sampling . The four procedures and concomitant 

restrictions are listed in Table 1 and an* example of each procedure is 

given in Table 2 for \k =*3 and K - 7. » 



Procedures 1, 2, and 3 are implemented easily in practice; Proc£&ure 

4, however, ,is more difficult and. the degree of difficulty increases 

' . • ' ' ' • °- 

with indreases in K. Within the .context df the design of experiments, 

Procedures 3 and 4 are referred to, respectively, as a "partially 

balanced incomplete Block" dejsign {PBIB) and a "balanced incomplete 

block" design (BIB) . That which is "partially balanced" or "balanced 11 • 

by each design is the item pa i rings . In the BIR design, all possible 

. * ~ . - 

item pairings occur among subtests and they occur with equal frequency; 

in the PBIB designs item pairings 5 do not occur with equal frequency and,. 

indeed, some item pairs may be excluded completely* A BIB design is 

*■ > • ' * • * 

often difficult to implement because, for a given, K, no design may 

— e * - 

exist, or, if there is a design, the number of subtests required is 

excessively .large. Tb^sf limitation is most serious when K exceeds 50 

even -permit ting minor adjustments, in K to fit in available design. For 

\ 

example, when K ^ 91 and k = 10, 91 subtests would be required; for 

#/t • 

K = 97 and \ = 10, N 4656; and", for K = 199 and k = 10, 19701. The firstf 
of these three BIB designs is cited and illustrated by Cochran and Cox 
(1957); the pUaer two are given by Ramanujacharyulu (1966,) and cited by 
Knapp (1968a). Although BIB designs have been used on a few occasions 
(e.g., Knapp,. 1968a, 1968b) when K was small, (K = 43 and K = 12, 
respectively, with Knapp), *such designs are ill-syited to large item 
, populations. This point is of no minor import because one of the major 

reasons for usirijf P multiple matrix s&mpling is ifrs potential for dealing 

> n 
wiCh large item populations. Because of this, it~*is expected Oat the 

' * t 

majority of item allocation procedures in multiple matrix sampling will 

* "* 

involve Procedure's 1, 2, ot 3. * 



■ -t 

It should be noted that, in practice,. Procedures 1,. 2, apd 3 are 
. implemented typically rnr conjunction with Item stratification, that is,* 

* e 

a stratified-random sampling procedure-is used with the stratification 
being on item content, item difficulty level, or both at; em content and 
item difficulty level. The relative merit^ of such stratification 
procedures have been discussed previously (i.e., Shoemaker and Osbutn, 
1968; Kleinke, 1971) and ar§ not 'considered here. . * ■* 

Of principal interest in this investigation were the relative 
merits of Procedures 1 and 3. 'Procedure 2 was excluded because it is 
used rarely \n practice. The metric by which these two item allocation 

procedures were contrasted was the standard ^error pf estimate, 

-» 

* ' METHOD 

< 

" * The research design was one* of* post mortem item-examinee sampling, 
<?with the required data bases generated through ^--crSmputer simulation 
model described previously by Shoemaker (1972). Jn post mortem Item- 
examinee sampling^ various samples of items and examinees t are selected 
randomly frotn a data base (an item by examinee matrix) and used to 
estimate parameters .o£ tjie base froin which they have been sampled. 'The 
researcher acts as if only certain examinees have been tested over 
certain items knowing all .the while the results- which were obtained by 
testing all Examinees over all items. 

Parameters^ of tfee data base manipulated systematically 'were:- 
(a) the number of test items (K - 40, 60), (b) variance of the item 
difficulty indices ,(cj « .00^ .05), .and (c) degree of skewness iri the 
normative distribution (distributed normally, markedly -negatively- skewed) . 



When the test scores were negatively- skewed, only- a.. — 0 was used.* The 

reliability of the total, scores */as equal to ,80 for all data bases 

generated. The 9 itemrexaminee sampling plans used are listed in , 

2 %t 

Table 3. A PBIB design was used only .when a > 0 for ,a given data base, 
v v - • ' P , * * 

2 1 . - - 

When % . := * 0, ^ tera s are statistically parallel and Procedures 1 and 

3 produce equivalent" results (and all differences observed between the 
two procedures would be due to the sampling of. examinees) . \ # 

The parameters estimated were ^ (the mean test sjcore), ^.-p^ 
(the second through' fourth central moments) and o\. The equations used 

to estimate the moments of the t.esti^s core distribution were those given 

2 * 

by Lord (1960); <? was estimated through a components ^of variance 
analysis. The results of each sampling plan were replicated 50 times. , 

Of additional concern in this investigation was a continued 
examination of the feasibility of -the jackknife procedure in approximating 
standard errors of estimate in* multiple matrix sampling, A description 
of ^Jhe "jaci&nife procedure and encouraging preliminary results in this 
area are^given by Shoemaker (1972). In general, the jackknife operates* 
on a data base which has been divided into subgroups of data and ^Ives 
a mean estimate of the parameter computed over subgroups and an estimate 
of the standard error of estimate associated with this statistic* A 
basic .component of the jackknife is the pseudovalue associated with each 
subgroup which, for each subgroup, is the weighted difference between 
the statistic^ computed on all the data and the statistic computed, on the 
body of data which remains after omitting that sutigroiip.' Because the" 
j)seudovaljues ££re relatively independent of each other, .the standard 
error of the statistic is computed according to the well-known formula 



for the standard error .of a sample mean. If the statistics computed oh 

each subgroup are weighted equally, the pseudovalues reduce algebraically 
■ „ *■ * " • * ■* 

to the averages for .the subgroups* When the jackknife is applied to 

^ ■ \ 

multiple matrix sampling there aire t subgroups of data but only one 



score (the estimated parameter) for each stlbgroup with that statistic 



weighted according to the number of observations J^k acquired by that ; 
suhtest* Ihe jackknife operates on the statistics obtained from one j 
$et of t subtests and approximates the variability of the pooled, estimates 
which would have been observed over repeated replications of the design, 

^ f ^ RESULTS ' 

AH' results are reported in Tables 3 through 7. Because the entries 

in each Table are interpreted similarly, only those for one sampling plan 

*' / 

in Table 3 will be described in detail. The first three entries in the 
first rpw of Table 3 .give the parameters of the data base. In this case, 
the IteLn population consisted of 40 it'ems, the' variance of the item^ 
difficulty indices (p = proportion answering the item correctly) was . * 
equal to 0 and the test scores were distributed normalfy. Using a 
(t = 4/1 =7 10/n « 50) item-examinee sampling'plan with random allocation 
of items, 'to subtests (Procedure 1 in Table 1) and replicating the Sampling" 
plan 50 :imes/ the standard deviation of the 50 pooled^estimates of the 
mean test score on the 40-item test was equal to .4695. Fifty jackknifed 
estimates of the standard error of the mean were produced. Their mean 

o » 

was equal to .4793; their standard deviation, .2445; If the items. for 

each .subtest had been allocated using a PBIB design (Procedure 3 in 

•» • * 

Table 1),* corresponding results would *have> appeared 1 under 'PBIB 1 in' the 



first row." None are giben there because a =0 and 1 the two itenv 

: p • V 

allocation^ procedures are* equivalent. 

Looking at the results !for SE(R) across Tables 3 through 7., it is 
generally the case that, for leach sampling plan, the standard error ,of » 

estimate is less when a. PBIB design is used,, The relative magnitude of 

* / - 

this .discrepancy was greater for the mean test score ancl decreased, 
sharply for* successively higher dentral moments , Jtecaus'd several 
combinations o£ Jt and k (for* a giVen tk) occurred among sampling plans }* * 
it was possible to .examine the" effect of certain combirjati6ns on t^ie v 

a * ^ 

• • v # 

standard error .of estimate. For a given Jtk, an increase in t* resulted 
in a decrease in SE(R) when estimating the mean test spote.; for^Jthe 

second through fourth central moments, an increase in k resulted in a 

* f * 1 " 2 * " 

decrease in SE(R); and, for -no trend was discernable. 

*' • 

Regarding the j-ackKnife v , the results indicate that on-t^e ave'ragg 

it did approximate well standard errors of , estimate. A major ^exception, 

* <* 

and one noted previously by Shoemaker (1972), was found in estimating 
the standard, error of the mean test score using a PBIB design where the* 
jackknife consistently and markedly overestimated SE(R). However, the 

jackknife did approximate well the standard error here when a random 

> 

sampling design was used to allocate items to^ subtests,* Looking at the 

i 

1 - 

results across parameters, it was generally found that, when a PBIB 
design was used, tlje > jackknife overestimated standard errors of estimate* 

This* did not occur when a random sampling design (Procedure 1 in Table 1) 

j m 
was used. The relative discrepancy was mbst marked for the mean test 

> 

^core and decreased in magnitude for successively higher central moments* 



In^a manner similar to SE(R), the standard deviation of the jackknifed 

estimates of the standard error SD(J) decreased with increases in^Jt when 

estimating the standard error of the mean test score and decreased m \ 

'« ' * > s 

generally with increases in k when estiriiating the 'standard errors* o£ 

the higher central moments for' a given tkv ^. 

< • : • * 

' * * * DISCISSION 



H * The results suppott trie conclusion that £h& procedure for allocating 

/ < 
items to subtests in multiple matrix sampling is ah important conSidera- ^ 

tias>n* Specifically, a partially balanced incomplete Block design is 

. ' - • * * * ■ / • ■* ' * 

preferably to a random allocation for sampling: plans having the same tfe # 

i * • 

- The superiority of the PB»IB*is most apparent in estimating the mean test 
score and becomes l6ss apparent in ^estimating higher central ^moments , 
This reinforces.>a conclusiqn made by Lord and Novick (1968) that in / 

• v / 

estimating the mean test score omitting even one item, has a drastic effect 
v ** + ' ' ° 

••#■•« • * 

on the standard error of estimate* In this investigation,, a PBIB design v 

guaranteed that each of the K itgms was included in some subtest. Such 

*» ■ . * * 

was *flot the case with a random, item allocation- where it yas quite 

- o • ? r 

possible for certain items 'to be omitted completely (.a's happened to 
item 2 in Procedure 1 in. Table >2) . The restilts indicate that the L<?rd 
.and Novick conclusion is applicable to higher central moments but the 
expected discrepancies are not as drastic as those expected with the 
mean test, score* , ' 4 

% *©f additional intfer^st in this investigation was the uss of the 
jackknife in approximating standard errors of estimate in multiple 
matrix sampling^ The results reinforce the conclusiqn ^drawn by 



Shoemaker (1972) th&t the jackknife can be used for this purpose and 
/ * » 

/ 

also £hed light bn a problem rilentioned therein* Shoemaker noted Chat 

. /*>■ '• .• • ' ' . 

the jackknife overestimated the standard ^rror of the mean test score > 

when a = .05 ani i^ems were allocated to subtests using a PBIB. design. 

w r . * \ ■ ' . 

_5h^resuJ.ts in £able3 suggest tha v t the inability* of the jackknife to 
pprforr^well frr^^ii^ case was a function of the item allocation procedure 

For the jackknife to be appropriate, the^pseudovalues must be independent 

• * . \* 

arid the results sugg^s,t that this requirement is violated with a PBIB 

design,. Regarding this violation, the jackknife is Dot as^ robust when 

estimating the standard error o*f the mean -test 'score as it is in 

est-iimatirtg standard* errors of higher central moments. The conclusion 

* + * 

*■ * 2 

seems warranted § that, ^hen'cr^ departs significantly from zero and ,a 

PBIB Resign is used tQ allocate items to subtests, the jackknife will 

approximate conservatively the standard 

matrix sampling* 



erro^ of estimate in multiple 



10 



. 9 



REFERENCES 



Cochran, W. G. & Cox,. G. Tfif Experimental designs . (2nd Edition) 
' New York: Wiley, 19574 . , - 

*Hooke, R. Symmetric functions, of a^ two-way array. 1 . Annals ? of * " • 
Mathematical S^tisti^s , ).956, 27, 55V79* \" " " 

"Kleinke, D. J» The-accuracy of estimated total test statistics^ 

* Final Report: Project No. IB07Q, 1 Grantf'JNb.* OEG-^7109^0, 
U..S. Department- of Health, E<Jycation, and Welfare, Office 
of Education, National, Center for Educational Research .and 
Development, 1972. t < . ' i 

Knapp, T, R. An application of balanced incomplete block designs to 

the estimation of test, norms. Educational and ^Psychological 
Measurement , 1968, 28, 265-272. * (a) / A 

Knapp, T. R. BIBD vs. PBIBD: ' an example of priori item sampling. 

Unpublished manuscript, University of Rochester,^ 1968 • (b) 



Lord-, F. M. Use. of truey^core theory 

and bivariate observed-scor& distributions 
< ' . 1960, 25, 325r342. " \ f 

' " ' ' ' \ JL- 

Lord, F. M. & Novick, M. R. Statistical tfheor 
v Reading, Mas?.; Acidison-Wesrey , 1968 



\ to predict moments of Univariate 
Psychbmetrika , 



ies of mental tesfc, scores « v 



Raraanujacharyulu, C. ^A new general series of ^balanced incomplete block 
designs . Proceedings of the American .Mathematical Society , 



1966, 17 3 



1064-1068/ \ : . 

j / / A ' , 

Shoemaker, D. M. & Osbfeth, H. GJ An' empirical, study of generalizability 

coefficients foi^ Unmatched dataA British Journal of 1 

Mathematical and Statistical* Psychology , 1968, 21, 23$-246. 

Shoemaker, D. M. A general procedure for -approximating .standard errors/ 
of estimate in multiple matrix sampling : Southwest Regional/ 
Laboratory for Educational Research and Development Technical 
Memorandum, 1972. . •* . ' \ 



ERIC 



11 



« . . TABLE 1 , I 

-I 1 • / ^ * 

Procedures for Allocating Items to Subtests in Multiple Matrix Sampling 



Item Allocation 
Procedure 



\ 



Restrictions On tk 



Resgy jctjbns Oh 

V Sampling Of Items 



!<►, Random Sampling 



Partially / 
Balanced j 
Incomplete / 
Block- Design 
(not all 'items 
tested) \ 

Partially- f 
Balanced 
Incomplete 
Block Design 
(all items 
tested) 



None 



tk < K 



tk ^£ 

tk =\rK (r 



integer) 



Wi-thout ,repracement 
within, ejach subtest 

With '^placement 
among Subtests # / 

Withqut replacement 
within each subtest 

"Without ( replacement 
, amoh^s ub tes t s 

Without replacement 
withi * each subtest 

Each/of tbe items 
appears with ecfiial 
frequency (r) among # - 
subtests 



Balanced 
Incomplete 
Block Design 



tk > K 

tk = rK (r integer) 

ClC k - 1 

(X integer) 



Without re vlacement 
within each subtest 

Each of the K(K - l)/2 
item pairings appears 
w^Lth equal frequency * 
among subtests 



11 



• / TABLE 2 

Examples of Subtests- Resulting From the Four Item.Allocation 
Procedures- Described in Table 1 Using k * 3* and K - 7> 



Subtest 
Number 



Pnggedure 1 f Procedure 2 Procedure 3 Procedure-4 





3 


5 * 


1 


(2 3 


1 


2 


3 


1 


2 


4 


2 


' . 3; 


4 


-5 


4 


'5 6 


4 


5 


6 


2 


3 


5 


? .« 1 


3 


•5 / 




« 

r 


-7- 


1> 


V 


- 3 


4 


6 




1 


4 


7 






3 


4 


5 


" 4 


5 


7 


5 


4 


5 


6 






• 6 


7 


1 


■ 5 


6 


i 


. 6 


I . . 3* 


4 


6 






2 


3 


4 


6 


7 


.2 


.7 


i ' 3 


'6' 


7 






'-5' 


6 


7 


7 


1 


3 



* \ 




12 



Joe 
5 



CO 

c 



vx 

© 
o 

O 



o 



o\ no aO 

to r^O u> , 

sj N H CM' 



C3 

I 

O 

a: 



o 

c/3 

o 



o\ o\ cc o> ON 
co •-< ^o <~ o 

N n O ■-• f-t 



CM CL 



o 

*\ 

N . 

00 
C 

srl 



tr 
c 

3 



W 




r<» n vo n co 

CO vO m H N 
^ CN H CO CM 



rN-'O con o 
el vo cm c\ \o 

<r vo cn o •-• 
co h c\ is n 



*-* O fH ^ in. 
CO ir» O <x O 
f-» m-O o 
o <o n tn n 



o i O co to co ^ O n cm vr> vo 

to cm 1-4 cm cn onroco , >j o mco vo 

m rs. cm, *-» oo(Mn h r^eo vo h o\ 

ION H ^ CM «-H O O 1-4 «-H © O i-i O 



/ 

n h n n n m •-• o\ co 

CO ^ ^3* vO CM- ' vO «-< ovco 

«-« to co >o mo n H t-* CM CO CO CM 

r- »-i c\ r-N. co co cm cm co >j co cm co co 



cc<n O h o cm co r^N von co 

o co ma < vtoomm n o <r h oo 

o> vo • <y co <omo. o O o cm to 

>HO\J C\ >J CO CM CO >J -CO CM <J- CO 



o o o o o 
to in co in to 



O Ot3 O 
to to to to 



© © © © o- 
toin to to to 



O O O vO CO - O O O <t OOOvOCO 

»— r-< »-t O < «— < i-i i-4 O i-H f— < i-H O 



3, CO CM O 
OC*-i •-» 



o 
o 



o 



o 
o 



o 

vO 



/ 



> 

CO CJ 

jo 

Seen 



.14 



13 



cs 

r-i 
&. 

CO 



c 



e 

O SO 
Z O 

C 9 
#-* -o 

o 

50 O 
G5 O 
€>• i* 



1. 



CO 



<3 ' 
U 

o 



S 
o 
*c 
c 



£ s 

03 r-< 
C , 

* z 

CM G. H 

° .« 

to 

M* ^ 

O 
* c 



c 



o 



r^ 
$ ft- 

c ^ 

r-i 
9 



U 

w 
« 

TS 

c / 
ea / 
u ' 
CO 




o co ^ in 

O O vO CM 

HO HCO 



o <o f» cv 

m CM <f rH 

O r» co vo 

CO iT» C% <y 



cv r» *-« co c\ 
cv«nn 
<r co oo r* <r 
co r* vo co co 



O <T \D vO «-i 

o <r r» co 

H CM CO 00 H 

ro if» co v\ <r 
• » • • • 
co cv co m o 



m #-» o 


o 


co 


CO 


Cv VO 


*^ r*» cm 


o 


Cv 


o 


cm <r 


«-» O cr> 


o 


<r 


r-i 


r>~ co 


h n r» o 


cv 


o 


r- 


r-i r-i 












N»y tnco 


OX 






•AO 

rH 



sr o o o 

CO «-« CO CM 

cv cm r>. m 

r* <X Cv'r-i 



cioco»a cm <* c» m o 

r- <r O mA h no 

Ov lO < CM CO CO H OV O 

•c cv «/* f"-. o m o vo .r*- 



m <7\ OV r*l CO 

r» o. o <y vo 

— « <J* <J\ CO vO 
r> r-t 6 Ch O 



in N h r» 


CO — i 


r-» CM 


CO 


vO 


CO r-» 


CM 












r-i 




•C «C~vO CO 


m o 


cv r* 


r-i 


r-i 


CO CV 


CM 


CO %C CO CO 


*co cv 


co <r 


CM 


CM 


r» cm 


CM 


m*ov r** «— « 


O vO 


CO f** 


Cv 


to 


CO vO 


CM 




Cv O 


r-i 




«c 


r-» m 


r-i 


• - * • • 












<• 


r-i CO O CO 


CO IA lA O 


CM 


CO 


*C CM 


Cv 


r-l CM 


. ^ 


CM 


r-i 






O O O CO 


t 

■CV-C0 


CM O 


CO 


r-i 


Cv CO 


CO 


o co r*» <y 


CO vO 


rH VO 


vO 


CO 


TM r-i 




r-^SM ^ CM 


m cm 


CO CO 


r* 


vO 


fs. VO 




CO <T TM-Q__ 


cv o 


co r* 


vO 


vO 


vO O 


<3 


o co r» m 




in c\ 


_ uvcvo~co 'o 


r-i CM 






CM 


r-i 


r-l CO 


r-i 



O r-i r-i r» CM CO r» in CO 

cm, cm r* cm m h m co co 

f>,»n m co vo m cv co cm 

Cv io o cm ca cm co <r vo 



I CO 



<T CM r-i <J> CM 



^ CM m O r-i 
r-i Cv vO CO vO 
CM CO CO m CM 

CO r>- co co »-» 
r-i "Cv CO vO f**- 



cm m O CM CM 
Cv Cv. CO O . 
Cv Cv CO O CO 
WVCO vO CO-CM-' 

cm o co vo r» 



cm r» <j co 

CO CO r-l r*. 
CO vO Cv vO 

o cm m <x 



o r* r* co co 

CV CM CM CM CM 

cm <r vo r>» co 
vo cm cv co m 

* cm Ov \o fV^ 

r-i r"l ' 



m co cm - r» 

,CM -O "v0 sO 
r-i Cv CO O 

•* • • • , 
CO in CO rM 



- -co co co r^-r* 
h n o r» <c 
in co co so 
«c r** O ^ 

o j>> S in r» 



o o o o 
m m vo m 

o o o g 

v* CO CM o 
O O r-i r-i 



o o o o 
m m m m 

O O O S3. 
r-T r-l r-i Cj 

SCO, CM O 
O lH r-l 



O O O O O 

m m irv m"ln 



O O O O CO 
r-l r-i r-i O r-i 



VO CM CO O O 
O «~» r-» r-l r-i 



o o.o o o 
in in in in in 



o o o *o CO 

i-t*r-i r-i O r-l 



dooo 
m in m in 

O O O <♦ ,000000 

rd-r-l r-i O — < »-* rH O r-i 



O CM CO O O or CO CM < 
O r-i r-l r-l r-l O O r-l I 



o 
o 



o 



o 

- vO 



cy 
> 

■h *a 

*j a 

«J ? 
co cj 

O -Si 

2 CO 



c4 



15 



14 



c 

*9 



Q. 

E 





O 


« 
-Vt. . 




^JE- 














Ci 






















s 


Cm, 










a. 


c 






o 




CO 








u 


m 


V-r 


S3 




O 


u 






O 




t> 


i~« 




ZJ 




c 


u 






CC 


£ 








Q 


a 










"D 


w 




- C 






< 


o 




CM c. 




















c 












CO 










o 






c _ 
o 










u 






o 






* c 












£• 






< 






CO 






< 


















V* 












Urn 












U 












G 
























W 






«-» 






O 





u 



«3 
C 



oc c 
c — 



CM CU 







« 








a 




g 
a 








CO 


> o 








u u 


o 






Si 


o 


H 


<y 




u 






« 


« 


c 


o 


h a 



co .£> *n co 
m O 

no NvO 
O N ^ 9 



00 © CO o 
*-» »r. cm 

O "» r 

O- vO 4T» 



CN «"■% CO vO < 



r. *•< 00 **> 

M M Q O 





" * 








«n 


CM 


*m 


0 


r*» 


«M 


irt 


CM 


CO 


r% 


•C 


0 


0 




x> 


«-H 






r*. 


CO 




3 


in 



m o» r* 


CM 


CO 


f>» 


cr- 


CM'tO' f-» r-4 


CM 




CCl 


0 






r> 


rr 


<o 


r* r- tn cr. 




00 


co 


r-« 










O C H O 




O 


<A 


0 


JT> r-» r-» vT 




cm 


#-< 







m 


W> 


0> 


n 00 




O 


O 




00 


co 


CO 


CM 


0 












N CM O 00 


CM 


cr 




co m <f ia 


00 


VA 





r* o f>co 

O <M h O 
<j n r> 



.*-« m in* e& 
ffv ir* nn 



C- 0> sa 
0>iTi fOM^O 

^ n >o ■ i 



«ft<f i-«<0 N 

O O <t c~« 

— t «o r- st> cr* 

o r> r*. 

On — t <^ cn 

CM <~t <** CM 



vD iJv CO 

^ f> iT-J rH 

«/"« cm r>* *fl <t 

<n u-> cr> cv 00 

CM r-l <M; 



41 N O vl ^ 

Jil M \fi r* -« 

ia vt <r *^ 

O f-^ 

< i -t» vT« CT 

o ,7 a-^ o <r 



<M vr w .-^> »" •- 
cb o 

OA r-^ %r\ O ^» 



S»-« O <J- C4 
<t CM »— • 
CO vO CTi in rv 
iTk ID \D «\ 

O fl O i/Ti 
CO *-* O 00 
H i-f r< f J 



O ^ CM O O 

co r> h in csi 
»-t in m m o 
HiniO\o ^ 

HO OlM> 
N H H « 



IN OlN H 

r^. <T CM CO 
m n a o 



H ^ H N 

no o o 
co o >n m 



H H IA Cf N 

co o C\ r» m 
cm Or* in m 
cr. 00 co eo"r-» 

■tn o o r**'va 
cm vo m cm co 



Hr-MOlTl 



m o *m o 



co c* co vo 

>J- CM CO O 
CM 0<» H 

' <r» <r *n \D 
•r> cm ^ 



vO n — < 

Ot CO CO -CM CO 

m o vo m vo 

CO M f» CO O 

co cm co co 
CO \0 CO *—» o 
CM »-< H n N 



o o 00 
m m m m 

o co ^ 

»-4 »-* *-* o 

S. ^ ^ ^^N. 

SCO CM O 
O HH 



OOOOO 

mm tn <n »n. 



O O O <J 
^ ^ ^ ^ 

SCO «Ni O 
O >-4 *-< 



OOOOO 
m in m in *n 

~"»» ^>>. J>> 

OOOM700 OOO CO 

rHi O «— • M H r* D H 

■Si ■»* V, ^s. 

^vOCMCOOO X CM CO O O 

O ^ HHH 0»-*»-it-*#-i 



o 
o 



O o o o 
m in m m 

o o O <t 00 

*~4 f-4 »-t O «-l l-l 

SCO CM O 
O 1-4 f-l 



OOOOO 

in m m m m 



o o o M0 co 
O n 



lO CM CO O O 

O »H f« H 



O 
M^ 



7^ 



CO 0) 
0 ^C 
^ CO 



ERIC 

Mffllffllffiffll'LH 



16 



15 



C3 

H , 

ca 



ih c/ O m 
m *a> O O* 

CN CN CM* ON 

m vo v<r cb 
r>» r>» r>» 

CO sf CN C 



CO CO CN ON 
ON ON O vO 

m r* co on 
<N o vo r> 

n h oo ov 
H v? o\«j 
«o h Ov<r 

HK VO 



t ON vO 
H 00 N «J 
<-*C0 H N 

on cm . 

ON O ■ 

co vo cn rs,*<y 
CN co 



i <f co n 
^ ^ i 

I 0> fN,*. 



vo n <t vo 
oo n rsvo n 
o^oS o 
co vn p co 

Cv »r vO»H 

on vn Ln vo.cs 
vo^oOnN 

<f CO CO CO CO 



c 

-to 

l-H 

a. 

CC C 

c 

h jm: 

rH 

B ^ 

C/5 



O ON CM CN 

CM cm-o r*. 
co m co co 
O <?;ON CO 



CO3D -cr O 
'CO rH O CO 
rH <t CO NO 
r> ; VO rH 

CO CO CO CM 

m on cn cm 
o O co r» 



ON CO OVrl 
O CO rH CO 
CM vO-fH rH 

n co 



m <t cv o 1 

O tr\ vO rH 1 
CO N MO IA 

«<r cm co vo 

0*r-» m co o 
vo <t *n vo rs 
co co o CO CO 
co cm 



on o O 



-co r*-co-com 
t-t o on co 

CO On CO CO *Ct 
O m in co 



O O O <J H 

m cm o cn <r 
no«j in co 
co rs rs rvK 



rH rH VO CO 
vO CM Cn 
vo On m cm 
O co m vo 



cm;co" 
' cm. m 



/ 



CO vo *ct 

CO ^ CM 

m n co 



CM NO O Ot 
rH CO CM VO 
CM VO H 

rH .cn 

m co o on 
cm m co 

CM OS <y CM 

m co co no 

vo on co 
vo ON cm sf 
co m in 

CM rH rH CO 



CO rH CO CO 

ON C.i CM O 



*-cn!, 



OV*rH r>- 

..on cm cm; 

O rH CM CM ; 
rM CM 



^ O 



O CM rH 1 
CM rH ON CO\ 
CO CM ON CM 



vo m ON <t 
VO CO CO CO 

r» o <vi on 

CM rH CO CO. 



CO ON ^ O CO 
mvONH^ 
<t vO Cv vj 
uS CN O rH 



NO rH'fs. CM ON 
CO CO rH ON 
.O CM rH ON CO 
CO CO O VO CM 



CO CO rH CM 

rs rs in n r^. 

O CO CM O ON 
CM rH rH VO 



O N •} O 
rs. rH r*. to co 
O fs CO Cv O 
fN. CM CM CM CO 



CO CN CO O 
CV CN H 
I O CO CO 



*Ct O <t CM 
CM O VO CO / 
VO CM rH NO 



^rH 

CO 



CO rH r-LCO-r^, 

cm vo co O 

NO CO VO »H VO 

' ~i fs u-i h n 



CO VO CM CO CO 

fx O ^ co m 
cm <t r>. co'^r 
m vo in. *cf 



<y <t vo vn vo 

rH vO CO » 
Ov<2- CO < 
VO CO ON CO rH 



<f ON-CO CM 

o O vn vn 
O m co co 
cm cm vn 



h fs 

H Pi O fN , 

ON rH m ON / 
rH *{|> CM O / 



co co 
co in 
ON/r* 



l H CO CM 

k no cm 

I CM NO <t/ 

m cVco 
^ O m 

1 ry m on 



/ 



CM 1 

o> l . 

rH O 
VO rH O 



H <t CO 
NO CO O 
ON r*. CN 
ON CM TO 



SO rH CO f>. 

CO co co 

CO CM ON CM 
CM CN H VO H H t <j- 



<t ro »c m 

VO,H ^ 



© cm r- vo co 
m <f -co cm cm 
on co ©• m CM 

o cv n <j- <r 

rH CM 



m m on *cf 

VY CO CO rH * 

m vo co vo co 

<t CO CO p CM^ 



NO CO CO rH 
CO rH VO *J 
rH CO CM O 

m co cm vo 



/ 



m <m V> <r o 
cm .co m <r O 

rH vO <f ON 



. cyo.o o 
vn An m vn 



0000 
m vn vn vn 



o © o <* O O O <f 

H H H O HHHO 

<f 00 O <t CO CM O 

OO H H OO HH 



o o o 0.0. 
m vn «n vn m 



O O o no co 

HH H O rl 



OOOOO 

vn vn to vn vn 



OO.O O O OOOO 

mmmmin mmmio 

000(0 00 000<f OOOvOCO 

H H^HO rH' H r!H O rHrHrHOr-4 



NO CM CO O O 
OHHHH 



VO CM" CO OO <f CO CM O 
OHHHH OOHH 



VO CM CO O O 
'O H H-H rH 















GO 






§ 








a» 


C 


CO 




0 






H 


VM> 


AJ 


u 


O 




3 








CJ 




H 


01 


O 


w 








CO 




CO 


SJ 


c 


H 


0 


H 


Q 



O 

9 



0 

NO 



m 
O 



O 

o 



o 



o 
o 



o 

vO 



I 

jj a 

co a 

<u ^ 

5S to 



I: 



16 



a 
s 
a 
to 



a>. 
to 

u 
o 

Dm, 



o 

2: 



SI 
CO 

IS, *w 



c 

o « 



t CI O 

i O rH 

t h , rH 

» 00 < 

a, s 

< S 



cm a 
o 

Q 



O 



c 



0. 



4» CS 

w > o 

<U *> 

° -fl 2 

B JO ' 

. « O U 

to « 
pwft 



O M 0\0 
in N H (O 

o o o*o 
o o o o 



oo r» <t vo 

o r*» vo n 

•rH o O h 

o o o o 



<r <j- co <n «h 

(OHONH 
O O O O O 

o o o o o 



0\ vO « _ 
O O O rH O 

ooooo 



\Q ifi vO 

o o o o 
o o o o 
o o oo. 



o o<o o 
o o o o 



00 N N OS 

Cn Csl rH CM 

o O o o 

o o o o 



m <T^ <T 

O O o o o 

O O O OSO 



CM -tf 00 r*. rH 
CO rH O CM rH 

ooooo 
o o o o o 



fs vo r» ts _ . . - 

oqoo d o o o o 

o oo o o p o o o 

oooo ooooo 



o co e» rH <j- r»- r>» 

rH. O O rH. 0\ p«» 0»rH 

OOOO O O Oh 

OOOO OOOO 



o\ co *o Ot o 
ooooo 
ooooo 
o o o o* o 



-d- as in. co r- 
co wv<r cn <r 
ooooo 
ooooo 



r- fs r- o 

O O O rH 

o o o o* 
o o o o 



in m m o m 
ooooo 
ooooo 
ooooo 



ON rH O CM 
O rt H H 

OOOO H O O rH 

OOOO oooo 



'H0£)O<t 
rH .CO CO CO 



rH O vO CM r- 

rH rH O rH O 
OOOOO 
O OOOO 



CM CO CM VS. 

o in <r co m 

rH O O O O 
OOOOO 



OOOO 
OOOO 



00 \D N CO 
OOOOO 

ooooo 
o o o*o o 



OOOO 

m m m m 



oooo 



•ooooo- 
m m m m in 



o o o o o 
m m m in m 



o O O <t o o o st 

rH rH'rH O HHHQ 

S3- CO <t CO CM O 

O O rH rH O O rH rH 



O OOvO CO oco«ooo 

r-t r-^ r-t O r-t rH rH rH O rH 



>0 CM CO O O 
O r rH H H H 



OOOO 

m m m m 

o o o <*. 

rH r-« rH O 

<t CO CM Q 
O O rH rH 



OOOOO 

m m m m m 



vO cm co o o 

' O r-t r-t r-t r-t 



O 

iP 



0) 
r> 

•H 73 
W 4) 
CO 3 

00 CJ 

01 ^fi 
^ CO 



DISTRIBUTION LIST 



% 

\ 

$ 


- Dr. Schutz 

./ 

- Dr/ Baker > 

/ " * 
-/Mr. Hein 






V 


• 






- Dr.. Berger 

- Dr. O'Hare' \ 










• 


1 


«~ Joan Midler " \ 




• 






* * 


1 

1 

' < 


\ 

- Library * 

- Archives * 1 






..... - # „ 


\ 





1^ - ABJ 
1 -*Dr. Hanson 
1 Dr. Ozenne 



15 - Dr. Shoemaker 



* H 



■■ t 



,ERJC 



19