/DOCUMENT* RESUME'
ED 110 519*
TM 00a 794
AUTHOE
TITLE
INSTITUTION
REPORT NO
PUB DATE
NOTE
Shoemaker/ David H. . -
& JLo*te-on allocating Items to Subtests in Hultiple
Matrix Sampling* .
"Southwest Regional Laboratory for Educational
P.esearch and Development, Los- Alatfitqs, Calif*
SWRL-TM-3-72-05
13 Jun 72 .
T9p.
EDBS PRICE
DESCRIPTORS
IDENTIFIERS
• {JF-$0.76 HO$i.58 PLUS POSTAGE < . (
*Iteta Sampling; *Matrices; ^Sampling; *Standard Errc:
"of Beasuremejit; ^Statistical' Analysis; Statistical
Bias; Testing Problems # *
Jackknife; *Multi"ple Matrix^Sampling
•ABSTRACT S f \^ m
Investigated' -empirically; through .post mortem ^
itemrexaainee sampling were t*hS relative merits of two. alternative
procedures for allocating items to subtests in multiple matrix
sampling a*nd the 'feasibility of using the jackknife in approximating,
standard errors. of esjtim.ate. The results indicate clearly that a
partially balanced incomplete block design is preferable to random
sampling in allocating items to subtests* The jackknife was found to.
-better approximate standard: errors of estimate in the "latter item
allocation procedure than in the former. .These and other results are
discussed in detail* (Author) - ,
\
* Documents acquired by ERIC include many infprmal unpublished *
* materials not available from .other sources/ ERIC makes every effort *
* to obtain the best copy available* nevertheless** 5,t* ms of marginal *
* reproducibility" are' often encountered and this affects the quality *
* of . the microfiche and hardcopy reproductions ERIC makes available *
* via the ERIC Document Reproduction Service (EDRS) ♦ EDRS&is not * *
* responsible for the quality of^the original document.* Reproductions *
* supplied by EDRS are the best that can # be made from the original* *
***************************** ********* **********************************
x—i
o
x — .
DATE
vn-
SWHL
;P£«Mi$$iON TO 'ft£PflCOUC£ T>-l$ COP*-
9»GH:ED UATEOVAl HAS 8£EN*GPANT£0 BY
SOUTHWEST REGIONAL. LABORATORY
TECHNICAL MEMORANDUM *
fOEBlC ASO ORGANIZATIONS OPERATING
WNCEH ACRKUEN^SWITM THE NATlONAt IN-
STATE OF EDUCATION PURSER PEPRO-
OUCTiCN OUTSIDE. ThE £«C . SYSTEM RE-
OWOES PERMtSStON 0^ THE COPYfcGMT
OVteNE 9
June t3, 1972 -
TM 3-72-05
U5 OEPARTMEMTOF HEALTH.
* EDUCATION* WELFARE
' NATIONAL INSTITUTE OF
EDUCATION
TrflS DC.-UWENr HAS BEEN REPRO
OUCSO EXACTLY RECEIVED FROAA
THE PERSCN OR ORGANIZATION ORIGIN
ATtNGIT POINTS OF VIEW OR OPINIONS
STATED DO NOT* NECESSARILY RSPRE
SENT OFFICIAL NATIONAL INSTITUTE OF
EDUCATION POSITION OR POLICY
TITLE: 7L NOTE ON ALLOCATING ITEMS TO SUBTESTS IN MULTIPLE MATRIX
SAMPLING . . f~
AUTHOR: David M» Shoemaker \
ABSTRACT
Investigated empirically through post mortem item*-examinee sampling
were the relative merits of two alternative procedures for allocating
items to subtests in multiple matrix s'ampling and the feasibility of
using the jackknife dn approximating standard errors of estimate. The
results indicate clearly that a partially balanced incomplete block
• • •
design is preferable to random sampling in allocating items to subtests.
The jackknife was found to better approximate standard errors of estimate
in the latter item allocation procedure than, in the former. These and
other results are discussed in detail. ' <
i
9
ERIC
p.tit docywent n intended to*. *.itcjmal staff distribution and use. Pernlsaion Co reprint or quote fr*?a thi* workirf K
docuaent. -holly or In part, should be obtained froa SWRL. 4665 LampSOn Ave • , LOS AjamitOS,.
A NOTE ON ALLOCATING ITEMS TO SUBTESTS IN MULTIPLE MATRIX SAMPLING
i ? «*
David .M« Shoemaker
. * ! " • •
Multiple matrix sampling or, more popularly:, item-exami it*e sampling,
»
is a procedure in which a set of K test items is subdivided randomly
>into Jt subtests containing k items each with each subtest administered
to n examinees selected randomly from the population of Re; 4iiinees«
Although each, examinee receives only a proportion of'fche K test "items,
the equations given by Hooke (1956) and Lord,, (1960) permit the, researcher
to estimate parameters of the test score distribution which would have
been obtained by tes tjlng all N examinees over .all K test items. Because
numerous combinations of _t, k, and n are feasible in any investigation,
the researcher. must coi&e to grips with several questions about how the
procedure should be implemented* "Ifow should items be allocated to
subtests? 11 , is one important question requiring an answer and is^the one
addressed specifically herein; concomitantly, the feasibility of using
the jackknife procedure for approximating standard errors of estimate
in multiple matrix sampling is considered .in some detail,
A basic requirement in multip'le matrix sampling is that k items
from the Kr item 'population are allacated randomly to- each subtest.
However, in constructing the Jt subtests , four, general item allocation
* < * - f . ♦
procedures are*BPSsible--each of which is described mone appropriately
as restricted random sampling . The four procedures and concomitant
restrictions are listed in Table 1 and an* example of each procedure is
given in Table 2 for \k =*3 and K - 7. »
Procedures 1, 2, and 3 are implemented easily in practice; Proc£&ure
4, however, ,is more difficult and. the degree of difficulty increases
' . • ' ' ' • °-
with indreases in K. Within the .context df the design of experiments,
Procedures 3 and 4 are referred to, respectively, as a "partially
balanced incomplete Block" dejsign {PBIB) and a "balanced incomplete
block" design (BIB) . That which is "partially balanced" or "balanced 11 •
by each design is the item pa i rings . In the BIR design, all possible
. * ~ . -
item pairings occur among subtests and they occur with equal frequency;
in the PBIB designs item pairings 5 do not occur with equal frequency and,.
indeed, some item pairs may be excluded completely* A BIB design is
*■ > • ' * • *
often difficult to implement because, for a given, K, no design may
— e * -
exist, or, if there is a design, the number of subtests required is
excessively .large. Tb^sf limitation is most serious when K exceeds 50
even -permit ting minor adjustments, in K to fit in available design. For
\
example, when K ^ 91 and k = 10, 91 subtests would be required; for
#/t •
K = 97 and \ = 10, N 4656; and", for K = 199 and k = 10, 19701. The firstf
of these three BIB designs is cited and illustrated by Cochran and Cox
(1957); the pUaer two are given by Ramanujacharyulu (1966,) and cited by
Knapp (1968a). Although BIB designs have been used on a few occasions
(e.g., Knapp,. 1968a, 1968b) when K was small, (K = 43 and K = 12,
respectively, with Knapp), *such designs are ill-syited to large item
, populations. This point is of no minor import because one of the major
reasons for usirijf P multiple matrix s&mpling is ifrs potential for dealing
> n
wiCh large item populations. Because of this, it~*is expected Oat the
' * t
majority of item allocation procedures in multiple matrix sampling will
* "*
involve Procedure's 1, 2, ot 3. *
■ -t
It should be noted that, in practice,. Procedures 1,. 2, apd 3 are
. implemented typically rnr conjunction with Item stratification, that is,*
* e
a stratified-random sampling procedure-is used with the stratification
being on item content, item difficulty level, or both at; em content and
item difficulty level. The relative merit^ of such stratification
procedures have been discussed previously (i.e., Shoemaker and Osbutn,
1968; Kleinke, 1971) and ar§ not 'considered here. . * ■*
Of principal interest in this investigation were the relative
merits of Procedures 1 and 3. 'Procedure 2 was excluded because it is
used rarely \n practice. The metric by which these two item allocation
procedures were contrasted was the standard ^error pf estimate,
-»
* ' METHOD
<
" * The research design was one* of* post mortem item-examinee sampling,
<?with the required data bases generated through ^--crSmputer simulation
model described previously by Shoemaker (1972). Jn post mortem Item-
examinee sampling^ various samples of items and examinees t are selected
randomly frotn a data base (an item by examinee matrix) and used to
estimate parameters .o£ tjie base froin which they have been sampled. 'The
researcher acts as if only certain examinees have been tested over
certain items knowing all .the while the results- which were obtained by
testing all Examinees over all items.
Parameters^ of tfee data base manipulated systematically 'were:-
(a) the number of test items (K - 40, 60), (b) variance of the item
difficulty indices ,(cj « .00^ .05), .and (c) degree of skewness iri the
normative distribution (distributed normally, markedly -negatively- skewed) .
When the test scores were negatively- skewed, only- a.. — 0 was used.* The
reliability of the total, scores */as equal to ,80 for all data bases
generated. The 9 itemrexaminee sampling plans used are listed in ,
2 %t
Table 3. A PBIB design was used only .when a > 0 for ,a given data base,
v v - • ' P , * *
2 1 . - -
When % . := * 0, ^ tera s are statistically parallel and Procedures 1 and
3 produce equivalent" results (and all differences observed between the
two procedures would be due to the sampling of. examinees) . \ #
The parameters estimated were ^ (the mean test sjcore), ^.-p^
(the second through' fourth central moments) and o\. The equations used
to estimate the moments of the t.esti^s core distribution were those given
2 *
by Lord (1960); <? was estimated through a components ^of variance
analysis. The results of each sampling plan were replicated 50 times. ,
Of additional concern in this investigation was a continued
examination of the feasibility of -the jackknife procedure in approximating
standard errors of estimate in* multiple matrix sampling, A description
of ^Jhe "jaci&nife procedure and encouraging preliminary results in this
area are^given by Shoemaker (1972). In general, the jackknife operates*
on a data base which has been divided into subgroups of data and ^Ives
a mean estimate of the parameter computed over subgroups and an estimate
of the standard error of estimate associated with this statistic* A
basic .component of the jackknife is the pseudovalue associated with each
subgroup which, for each subgroup, is the weighted difference between
the statistic^ computed on all the data and the statistic computed, on the
body of data which remains after omitting that sutigroiip.' Because the"
j)seudovaljues ££re relatively independent of each other, .the standard
error of the statistic is computed according to the well-known formula
for the standard error .of a sample mean. If the statistics computed oh
each subgroup are weighted equally, the pseudovalues reduce algebraically
■ „ *■ * " • * ■*
to the averages for .the subgroups* When the jackknife is applied to
^ ■ \
multiple matrix sampling there aire t subgroups of data but only one
score (the estimated parameter) for each stlbgroup with that statistic
weighted according to the number of observations J^k acquired by that ;
suhtest* Ihe jackknife operates on the statistics obtained from one j
$et of t subtests and approximates the variability of the pooled, estimates
which would have been observed over repeated replications of the design,
^ f ^ RESULTS '
AH' results are reported in Tables 3 through 7. Because the entries
in each Table are interpreted similarly, only those for one sampling plan
*' /
in Table 3 will be described in detail. The first three entries in the
first rpw of Table 3 .give the parameters of the data base. In this case,
the IteLn population consisted of 40 it'ems, the' variance of the item^
difficulty indices (p = proportion answering the item correctly) was . *
equal to 0 and the test scores were distributed normalfy. Using a
(t = 4/1 =7 10/n « 50) item-examinee sampling'plan with random allocation
of items, 'to subtests (Procedure 1 in Table 1) and replicating the Sampling"
plan 50 :imes/ the standard deviation of the 50 pooled^estimates of the
mean test score on the 40-item test was equal to .4695. Fifty jackknifed
estimates of the standard error of the mean were produced. Their mean
o »
was equal to .4793; their standard deviation, .2445; If the items. for
each .subtest had been allocated using a PBIB design (Procedure 3 in
•» • *
Table 1),* corresponding results would *have> appeared 1 under 'PBIB 1 in' the
first row." None are giben there because a =0 and 1 the two itenv
: p • V
allocation^ procedures are* equivalent.
Looking at the results !for SE(R) across Tables 3 through 7., it is
generally the case that, for leach sampling plan, the standard error ,of »
estimate is less when a. PBIB design is used,, The relative magnitude of
* / -
this .discrepancy was greater for the mean test score ancl decreased,
sharply for* successively higher dentral moments , Jtecaus'd several
combinations o£ Jt and k (for* a giVen tk) occurred among sampling plans }* *
it was possible to .examine the" effect of certain combirjati6ns on t^ie v
a * ^
• • v #
standard error .of estimate. For a given Jtk, an increase in t* resulted
in a decrease in SE(R) when estimating the mean test spote.; for^Jthe
second through fourth central moments, an increase in k resulted in a
* f * 1 " 2 * "
decrease in SE(R); and, for -no trend was discernable.
*' •
Regarding the j-ackKnife v , the results indicate that on-t^e ave'ragg
it did approximate well standard errors of , estimate. A major ^exception,
* <*
and one noted previously by Shoemaker (1972), was found in estimating
the standard, error of the mean test score using a PBIB design where the*
jackknife consistently and markedly overestimated SE(R). However, the
jackknife did approximate well the standard error here when a random
>
sampling design was used to allocate items to^ subtests,* Looking at the
i
1 -
results across parameters, it was generally found that, when a PBIB
design was used, tlje > jackknife overestimated standard errors of estimate*
This* did not occur when a random sampling design (Procedure 1 in Table 1)
j m
was used. The relative discrepancy was mbst marked for the mean test
>
^core and decreased in magnitude for successively higher central moments*
In^a manner similar to SE(R), the standard deviation of the jackknifed
estimates of the standard error SD(J) decreased with increases in^Jt when
estimating the standard error of the mean test score and decreased m \
'« ' * > s
generally with increases in k when estiriiating the 'standard errors* o£
the higher central moments for' a given tkv ^.
< • : • *
' * * * DISCISSION
H * The results suppott trie conclusion that £h& procedure for allocating
/ <
items to subtests in multiple matrix sampling is ah important conSidera- ^
tias>n* Specifically, a partially balanced incomplete Block design is
. ' - • * * * ■ / • ■* ' *
preferably to a random allocation for sampling: plans having the same tfe #
i * •
- The superiority of the PB»IB*is most apparent in estimating the mean test
score and becomes l6ss apparent in ^estimating higher central ^moments ,
This reinforces.>a conclusiqn made by Lord and Novick (1968) that in /
• v /
estimating the mean test score omitting even one item, has a drastic effect
v ** + ' ' °
••#■•« • *
on the standard error of estimate* In this investigation,, a PBIB design v
guaranteed that each of the K itgms was included in some subtest. Such
*» ■ . * *
was *flot the case with a random, item allocation- where it yas quite
- o • ? r
possible for certain items 'to be omitted completely (.a's happened to
item 2 in Procedure 1 in. Table >2) . The restilts indicate that the L<?rd
.and Novick conclusion is applicable to higher central moments but the
expected discrepancies are not as drastic as those expected with the
mean test, score* , ' 4
% *©f additional intfer^st in this investigation was the uss of the
jackknife in approximating standard errors of estimate in multiple
matrix sampling^ The results reinforce the conclusiqn ^drawn by
Shoemaker (1972) th&t the jackknife can be used for this purpose and
/ * »
/
also £hed light bn a problem rilentioned therein* Shoemaker noted Chat
. /*>■ '• .• • ' ' .
the jackknife overestimated the standard ^rror of the mean test score >
when a = .05 ani i^ems were allocated to subtests using a PBIB. design.
w r . * \ ■ ' .
_5h^resuJ.ts in £able3 suggest tha v t the inability* of the jackknife to
pprforr^well frr^^ii^ case was a function of the item allocation procedure
For the jackknife to be appropriate, the^pseudovalues must be independent
• * . \*
arid the results sugg^s,t that this requirement is violated with a PBIB
design,. Regarding this violation, the jackknife is Dot as^ robust when
estimating the standard error o*f the mean -test 'score as it is in
est-iimatirtg standard* errors of higher central moments. The conclusion
* + *
*■ * 2
seems warranted § that, ^hen'cr^ departs significantly from zero and ,a
PBIB Resign is used tQ allocate items to subtests, the jackknife will
approximate conservatively the standard
matrix sampling*
erro^ of estimate in multiple
10
. 9
REFERENCES
Cochran, W. G. & Cox,. G. Tfif Experimental designs . (2nd Edition)
' New York: Wiley, 19574 . , -
*Hooke, R. Symmetric functions, of a^ two-way array. 1 . Annals ? of * " •
Mathematical S^tisti^s , ).956, 27, 55V79* \" " "
"Kleinke, D. J» The-accuracy of estimated total test statistics^
* Final Report: Project No. IB07Q, 1 Grantf'JNb.* OEG-^7109^0,
U..S. Department- of Health, E<Jycation, and Welfare, Office
of Education, National, Center for Educational Research .and
Development, 1972. t < . ' i
Knapp, T, R. An application of balanced incomplete block designs to
the estimation of test, norms. Educational and ^Psychological
Measurement , 1968, 28, 265-272. * (a) / A
Knapp, T. R. BIBD vs. PBIBD: ' an example of priori item sampling.
Unpublished manuscript, University of Rochester,^ 1968 • (b)
Lord-, F. M. Use. of truey^core theory
and bivariate observed-scor& distributions
< ' . 1960, 25, 325r342. " \ f
' " ' ' ' \ JL-
Lord, F. M. & Novick, M. R. Statistical tfheor
v Reading, Mas?.; Acidison-Wesrey , 1968
\ to predict moments of Univariate
Psychbmetrika ,
ies of mental tesfc, scores « v
Raraanujacharyulu, C. ^A new general series of ^balanced incomplete block
designs . Proceedings of the American .Mathematical Society ,
1966, 17 3
1064-1068/ \ : .
j / / A ' ,
Shoemaker, D. M. & Osbfeth, H. GJ An' empirical, study of generalizability
coefficients foi^ Unmatched dataA British Journal of 1
Mathematical and Statistical* Psychology , 1968, 21, 23$-246.
Shoemaker, D. M. A general procedure for -approximating .standard errors/
of estimate in multiple matrix sampling : Southwest Regional/
Laboratory for Educational Research and Development Technical
Memorandum, 1972. . •* . ' \
ERIC
11
« . . TABLE 1 , I
-I 1 • / ^ *
Procedures for Allocating Items to Subtests in Multiple Matrix Sampling
Item Allocation
Procedure
\
Restrictions On tk
Resgy jctjbns Oh
V Sampling Of Items
!<►, Random Sampling
Partially /
Balanced j
Incomplete /
Block- Design
(not all 'items
tested) \
Partially- f
Balanced
Incomplete
Block Design
(all items
tested)
None
tk < K
tk ^£
tk =\rK (r
integer)
Wi-thout ,repracement
within, ejach subtest
With '^placement
among Subtests # /
Withqut replacement
within each subtest
"Without ( replacement
, amoh^s ub tes t s
Without replacement
withi * each subtest
Each/of tbe items
appears with ecfiial
frequency (r) among # -
subtests
Balanced
Incomplete
Block Design
tk > K
tk = rK (r integer)
ClC k - 1
(X integer)
Without re vlacement
within each subtest
Each of the K(K - l)/2
item pairings appears
w^Lth equal frequency *
among subtests
11
• / TABLE 2
Examples of Subtests- Resulting From the Four Item.Allocation
Procedures- Described in Table 1 Using k * 3* and K - 7>
Subtest
Number
Pnggedure 1 f Procedure 2 Procedure 3 Procedure-4
3
5 *
1
(2 3
1
2
3
1
2
4
2
' . 3;
4
-5
4
'5 6
4
5
6
2
3
5
? .« 1
3
•5 /
«
r
-7-
1>
V
- 3
4
6
1
4
7
3
4
5
" 4
5
7
5
4
5
6
• 6
7
1
■ 5
6
i
. 6
I . . 3*
4
6
2
3
4
6
7
.2
.7
i ' 3
'6'
7
'-5'
6
7
7
1
3
* \
12
Joe
5
CO
c
vx
©
o
O
o
o\ no aO
to r^O u> ,
sj N H CM'
C3
I
O
a:
o
c/3
o
o\ o\ cc o> ON
co •-< ^o <~ o
N n O ■-• f-t
CM CL
o
*\
N .
00
C
srl
tr
c
3
W
r<» n vo n co
CO vO m H N
^ CN H CO CM
rN-'O con o
el vo cm c\ \o
<r vo cn o •-•
co h c\ is n
*-* O fH ^ in.
CO ir» O <x O
f-» m-O o
o <o n tn n
o i O co to co ^ O n cm vr> vo
to cm 1-4 cm cn onroco , >j o mco vo
m rs. cm, *-» oo(Mn h r^eo vo h o\
ION H ^ CM «-H O O 1-4 «-H © O i-i O
/
n h n n n m •-• o\ co
CO ^ ^3* vO CM- ' vO «-< ovco
«-« to co >o mo n H t-* CM CO CO CM
r- »-i c\ r-N. co co cm cm co >j co cm co co
cc<n O h o cm co r^N von co
o co ma < vtoomm n o <r h oo
o> vo • <y co <omo. o O o cm to
>HO\J C\ >J CO CM CO >J -CO CM <J- CO
o o o o o
to in co in to
O Ot3 O
to to to to
© © © © o-
toin to to to
O O O vO CO - O O O <t OOOvOCO
»— r-< »-t O < «— < i-i i-4 O i-H f— < i-H O
3, CO CM O
OC*-i •-»
o
o
o
o
o
o
vO
/
>
CO CJ
jo
Seen
.14
13
cs
r-i
&.
CO
c
e
O SO
Z O
C 9
#-* -o
o
50 O
G5 O
€>• i*
1.
CO
<3 '
U
o
S
o
*c
c
£ s
03 r-<
C ,
* z
CM G. H
° .«
to
M* ^
O
* c
c
o
r^
$ ft-
c ^
r-i
9
U
w
«
TS
c /
ea /
u '
CO
o co ^ in
O O vO CM
HO HCO
o <o f» cv
m CM <f rH
O r» co vo
CO iT» C% <y
cv r» *-« co c\
cv«nn
<r co oo r* <r
co r* vo co co
O <T \D vO «-i
o <r r» co
H CM CO 00 H
ro if» co v\ <r
• » • • •
co cv co m o
m #-» o
o
co
CO
Cv VO
*^ r*» cm
o
Cv
o
cm <r
«-» O cr>
o
<r
r-i
r>~ co
h n r» o
cv
o
r-
r-i r-i
N»y tnco
OX
•AO
rH
sr o o o
CO «-« CO CM
cv cm r>. m
r* <X Cv'r-i
cioco»a cm <* c» m o
r- <r O mA h no
Ov lO < CM CO CO H OV O
•c cv «/* f"-. o m o vo .r*-
m <7\ OV r*l CO
r» o. o <y vo
— « <J* <J\ CO vO
r> r-t 6 Ch O
in N h r»
CO — i
r-» CM
CO
vO
CO r-»
CM
r-i
•C «C~vO CO
m o
cv r*
r-i
r-i
CO CV
CM
CO %C CO CO
*co cv
co <r
CM
CM
r» cm
CM
m*ov r** «— «
O vO
CO f**
Cv
to
CO vO
CM
Cv O
r-i
«c
r-» m
r-i
• - * • •
<•
r-i CO O CO
CO IA lA O
CM
CO
*C CM
Cv
r-l CM
. ^
CM
r-i
O O O CO
t
■CV-C0
CM O
CO
r-i
Cv CO
CO
o co r*» <y
CO vO
rH VO
vO
CO
TM r-i
r-^SM ^ CM
m cm
CO CO
r*
vO
fs. VO
CO <T TM-Q__
cv o
co r*
vO
vO
vO O
<3
o co r» m
in c\
_ uvcvo~co 'o
r-i CM
CM
r-i
r-l CO
r-i
O r-i r-i r» CM CO r» in CO
cm, cm r* cm m h m co co
f>,»n m co vo m cv co cm
Cv io o cm ca cm co <r vo
I CO
<T CM r-i <J> CM
^ CM m O r-i
r-i Cv vO CO vO
CM CO CO m CM
CO r>- co co »-»
r-i "Cv CO vO f**-
cm m O CM CM
Cv Cv. CO O .
Cv Cv CO O CO
WVCO vO CO-CM-'
cm o co vo r»
cm r» <j co
CO CO r-l r*.
CO vO Cv vO
o cm m <x
o r* r* co co
CV CM CM CM CM
cm <r vo r>» co
vo cm cv co m
* cm Ov \o fV^
r-i r"l '
m co cm - r»
,CM -O "v0 sO
r-i Cv CO O
•* • • • ,
CO in CO rM
- -co co co r^-r*
h n o r» <c
in co co so
«c r** O ^
o j>> S in r»
o o o o
m m vo m
o o o g
v* CO CM o
O O r-i r-i
o o o o
m m m m
O O O S3.
r-T r-l r-i Cj
SCO, CM O
O lH r-l
O O O O O
m m irv m"ln
O O O O CO
r-l r-i r-i O r-i
VO CM CO O O
O «~» r-» r-l r-i
o o.o o o
in in in in in
o o o *o CO
i-t*r-i r-i O r-l
dooo
m in m in
O O O <♦ ,000000
rd-r-l r-i O — < »-* rH O r-i
O CM CO O O or CO CM <
O r-i r-l r-l r-l O O r-l I
o
o
o
o
- vO
cy
>
■h *a
*j a
«J ?
co cj
O -Si
2 CO
c4
15
14
c
*9
Q.
E
O
«
-Vt. .
^JE-
Ci
s
Cm,
a.
c
o
CO
u
m
V-r
S3
O
u
O
t>
i~«
ZJ
c
u
CC
£
Q
a
"D
w
- C
<
o
CM c.
c
CO
o
c _
o
u
o
* c
£•
<
CO
<
V*
Urn
U
G
W
«-»
O
u
«3
C
oc c
c —
CM CU
«
a
g
a
CO
> o
u u
o
Si
o
H
<y
u
«
«
c
o
h a
co .£> *n co
m O
no NvO
O N ^ 9
00 © CO o
*-» »r. cm
O "» r
O- vO 4T»
CN «"■% CO vO <
r. *•< 00 **>
M M Q O
" *
«n
CM
*m
0
r*»
«M
irt
CM
CO
r%
•C
0
0
x>
«-H
r*.
CO
3
in
m o» r*
CM
CO
f>»
cr-
CM'tO' f-» r-4
CM
CCl
0
r>
rr
<o
r* r- tn cr.
00
co
r-«
O C H O
O
<A
0
JT> r-» r-» vT
cm
#-<
m
W>
0>
n 00
O
O
00
co
CO
CM
0
N CM O 00
CM
cr
co m <f ia
00
VA
r* o f>co
O <M h O
<j n r>
.*-« m in* e&
ffv ir* nn
C- 0> sa
0>iTi fOM^O
^ n >o ■ i
«ft<f i-«<0 N
O O <t c~«
— t «o r- st> cr*
o r> r*.
On — t <^ cn
CM <~t <** CM
vD iJv CO
^ f> iT-J rH
«/"« cm r>* *fl <t
<n u-> cr> cv 00
CM r-l <M;
41 N O vl ^
Jil M \fi r* -«
ia vt <r *^
O f-^
< i -t» vT« CT
o ,7 a-^ o <r
<M vr w .-^> »" •-
cb o
OA r-^ %r\ O ^»
S»-« O <J- C4
<t CM »— •
CO vO CTi in rv
iTk ID \D «\
O fl O i/Ti
CO *-* O 00
H i-f r< f J
O ^ CM O O
co r> h in csi
»-t in m m o
HiniO\o ^
HO OlM>
N H H «
IN OlN H
r^. <T CM CO
m n a o
H ^ H N
no o o
co o >n m
H H IA Cf N
co o C\ r» m
cm Or* in m
cr. 00 co eo"r-»
■tn o o r**'va
cm vo m cm co
Hr-MOlTl
m o *m o
co c* co vo
>J- CM CO O
CM 0<» H
' <r» <r *n \D
•r> cm ^
vO n — <
Ot CO CO -CM CO
m o vo m vo
CO M f» CO O
co cm co co
CO \0 CO *—» o
CM »-< H n N
o o 00
m m m m
o co ^
»-4 »-* *-* o
S. ^ ^ ^^N.
SCO CM O
O HH
OOOOO
mm tn <n »n.
O O O <J
^ ^ ^ ^
SCO «Ni O
O >-4 *-<
OOOOO
m in m in *n
~"»» ^>>. J>>
OOOM700 OOO CO
rHi O «— • M H r* D H
■Si ■»* V, ^s.
^vOCMCOOO X CM CO O O
O ^ HHH 0»-*»-it-*#-i
o
o
O o o o
m in m m
o o O <t 00
*~4 f-4 »-t O «-l l-l
SCO CM O
O 1-4 f-l
OOOOO
in m m m m
o o o M0 co
O n
lO CM CO O O
O »H f« H
O
M^
7^
CO 0)
0 ^C
^ CO
ERIC
Mffllffllffiffll'LH
16
15
C3
H ,
ca
ih c/ O m
m *a> O O*
CN CN CM* ON
m vo v<r cb
r>» r>» r>»
CO sf CN C
CO CO CN ON
ON ON O vO
m r* co on
<N o vo r>
n h oo ov
H v? o\«j
«o h Ov<r
HK VO
t ON vO
H 00 N «J
<-*C0 H N
on cm .
ON O ■
co vo cn rs,*<y
CN co
i <f co n
^ ^ i
I 0> fN,*.
vo n <t vo
oo n rsvo n
o^oS o
co vn p co
Cv »r vO»H
on vn Ln vo.cs
vo^oOnN
<f CO CO CO CO
c
-to
l-H
a.
CC C
c
h jm:
rH
B ^
C/5
O ON CM CN
CM cm-o r*.
co m co co
O <?;ON CO
CO3D -cr O
'CO rH O CO
rH <t CO NO
r> ; VO rH
CO CO CO CM
m on cn cm
o O co r»
ON CO OVrl
O CO rH CO
CM vO-fH rH
n co
m <t cv o 1
O tr\ vO rH 1
CO N MO IA
«<r cm co vo
0*r-» m co o
vo <t *n vo rs
co co o CO CO
co cm
on o O
-co r*-co-com
t-t o on co
CO On CO CO *Ct
O m in co
O O O <J H
m cm o cn <r
no«j in co
co rs rs rvK
rH rH VO CO
vO CM Cn
vo On m cm
O co m vo
cm;co"
' cm. m
/
CO vo *ct
CO ^ CM
m n co
CM NO O Ot
rH CO CM VO
CM VO H
rH .cn
m co o on
cm m co
CM OS <y CM
m co co no
vo on co
vo ON cm sf
co m in
CM rH rH CO
CO rH CO CO
ON C.i CM O
*-cn!,
OV*rH r>-
..on cm cm;
O rH CM CM ;
rM CM
^ O
O CM rH 1
CM rH ON CO\
CO CM ON CM
vo m ON <t
VO CO CO CO
r» o <vi on
CM rH CO CO.
CO ON ^ O CO
mvONH^
<t vO Cv vj
uS CN O rH
NO rH'fs. CM ON
CO CO rH ON
.O CM rH ON CO
CO CO O VO CM
CO CO rH CM
rs rs in n r^.
O CO CM O ON
CM rH rH VO
O N •} O
rs. rH r*. to co
O fs CO Cv O
fN. CM CM CM CO
CO CN CO O
CV CN H
I O CO CO
*Ct O <t CM
CM O VO CO /
VO CM rH NO
^rH
CO
CO rH r-LCO-r^,
cm vo co O
NO CO VO »H VO
' ~i fs u-i h n
CO VO CM CO CO
fx O ^ co m
cm <t r>. co'^r
m vo in. *cf
<y <t vo vn vo
rH vO CO »
Ov<2- CO <
VO CO ON CO rH
<f ON-CO CM
o O vn vn
O m co co
cm cm vn
h fs
H Pi O fN ,
ON rH m ON /
rH *{|> CM O /
co co
co in
ON/r*
l H CO CM
k no cm
I CM NO <t/
m cVco
^ O m
1 ry m on
/
CM 1
o> l .
rH O
VO rH O
H <t CO
NO CO O
ON r*. CN
ON CM TO
SO rH CO f>.
CO co co
CO CM ON CM
CM CN H VO H H t <j-
<t ro »c m
VO,H ^
© cm r- vo co
m <f -co cm cm
on co ©• m CM
o cv n <j- <r
rH CM
m m on *cf
VY CO CO rH *
m vo co vo co
<t CO CO p CM^
NO CO CO rH
CO rH VO *J
rH CO CM O
m co cm vo
/
m <m V> <r o
cm .co m <r O
rH vO <f ON
. cyo.o o
vn An m vn
0000
m vn vn vn
o © o <* O O O <f
H H H O HHHO
<f 00 O <t CO CM O
OO H H OO HH
o o o 0.0.
m vn «n vn m
O O o no co
HH H O rl
OOOOO
vn vn to vn vn
OO.O O O OOOO
mmmmin mmmio
000(0 00 000<f OOOvOCO
H H^HO rH' H r!H O rHrHrHOr-4
NO CM CO O O
OHHHH
VO CM" CO OO <f CO CM O
OHHHH OOHH
VO CM CO O O
'O H H-H rH
GO
§
a»
C
CO
0
H
VM>
AJ
u
O
3
CJ
H
01
O
w
CO
CO
SJ
c
H
0
H
Q
O
9
0
NO
m
O
O
o
o
o
o
o
vO
I
jj a
co a
<u ^
5S to
I:
16
a
s
a
to
a>.
to
u
o
Dm,
o
2:
SI
CO
IS, *w
c
o «
t CI O
i O rH
t h , rH
» 00 <
a, s
< S
cm a
o
Q
O
c
0.
4» CS
w > o
<U *>
° -fl 2
B JO '
. « O U
to «
pwft
O M 0\0
in N H (O
o o o*o
o o o o
oo r» <t vo
o r*» vo n
•rH o O h
o o o o
<r <j- co <n «h
(OHONH
O O O O O
o o o o o
0\ vO « _
O O O rH O
ooooo
\Q ifi vO
o o o o
o o o o
o o oo.
o o<o o
o o o o
00 N N OS
Cn Csl rH CM
o O o o
o o o o
m <T^ <T
O O o o o
O O O OSO
CM -tf 00 r*. rH
CO rH O CM rH
ooooo
o o o o o
fs vo r» ts _ . . -
oqoo d o o o o
o oo o o p o o o
oooo ooooo
o co e» rH <j- r»- r>»
rH. O O rH. 0\ p«» 0»rH
OOOO O O Oh
OOOO OOOO
o\ co *o Ot o
ooooo
ooooo
o o o o* o
-d- as in. co r-
co wv<r cn <r
ooooo
ooooo
r- fs r- o
O O O rH
o o o o*
o o o o
in m m o m
ooooo
ooooo
ooooo
ON rH O CM
O rt H H
OOOO H O O rH
OOOO oooo
'H0£)O<t
rH .CO CO CO
rH O vO CM r-
rH rH O rH O
OOOOO
O OOOO
CM CO CM VS.
o in <r co m
rH O O O O
OOOOO
OOOO
OOOO
00 \D N CO
OOOOO
ooooo
o o o*o o
OOOO
m m m m
oooo
•ooooo-
m m m m in
o o o o o
m m m in m
o O O <t o o o st
rH rH'rH O HHHQ
S3- CO <t CO CM O
O O rH rH O O rH rH
O OOvO CO oco«ooo
r-t r-^ r-t O r-t rH rH rH O rH
>0 CM CO O O
O r rH H H H
OOOO
m m m m
o o o <*.
rH r-« rH O
<t CO CM Q
O O rH rH
OOOOO
m m m m m
vO cm co o o
' O r-t r-t r-t r-t
O
iP
0)
r>
•H 73
W 4)
CO 3
00 CJ
01 ^fi
^ CO
DISTRIBUTION LIST
%
\
$
- Dr. Schutz
./
- Dr/ Baker >
/ " *
-/Mr. Hein
V
•
- Dr.. Berger
- Dr. O'Hare' \
•
1
«~ Joan Midler " \
•
* *
1
1
' <
\
- Library *
- Archives * 1
..... - # „
\
1^ - ABJ
1 -*Dr. Hanson
1 Dr. Ozenne
15 - Dr. Shoemaker
* H
■■ t
,ERJC
19
