Davis Report - Findings and Analysis
More Student Learning, Less Faculty Work? The WPI Davis Experiment in Educational Quality and Productivity
VI. FINDINGS AND ANALYSIS
- Impact on Students in PAC Courses
Reports from individual course initiatives were subjectively useful, but data were collected and reported in highly variable and inconsistent formats. Individual course reports were summarized earlier. Common themes emerging from the individual reports of the disciplinary initiatives are the following. Not every result noted was reported for every initiative, but overall the consensus was that peer assisted CL results in:
- improved student performance;
- improved student learning, especially in areas which transcend factual knowledge. Such areas include the higher levels of Bloom's taxonomy, oral and written communication skills, the ability to find information on their own, and teamwork;
- mixed student satisfaction with the course and the professor. That is, comparison of the PAC and traditional formats yielded significantly better, significantly worse, or insignificantly different student ratings of different course pairs, and even of different PAC offerings of the same course;
- mixed attitudes toward group work;
- the same or less faculty time input (in courses other than math
courses).
- careful selection, based on prior course success and interpersonal skills;
- proper training, usually consisting of 10 hours before the course starts. Considerable emphasis was placed on group dynamics in the non-math courses, and on course content in the math courses;
- continuing support in the form of weekly staff meetings and frequent email communication between faculty and PLAs;
- tasks for student teams that are carefully structured by the course
professor. PLA-led group sessions should always be structured with a
clear objective and means to reach it.
- Impact on Student Performance in 3rd and 4th Year Courses
- PAC Impact on Grades
Table 4 summarizes our main findings concerning the effects of PAC course taking on student grades. This table of key statistics from numerous regression models is composed in a short-hand that requires some elaboration.
There are 17 regressions summarized in Table 4. We describe them with reference to successive columns of information, starting at the left side. Regressions 1 through 14 were done for all students in the Class of 1997, excluding transfer students. There were 336 students in this data base. Regressions 15 through 17 were done for students whose high school class rank exceeded the 89th percentile. This identified the 56 percent of the class that might be considered "more prepared" for college level academic work. Interestingly, similar regressions for students whose high school rank was less than the 90th percentile--the "less prepared" 44 percent of our sample-yielded no statistically significant results, and thus are not reported here.
Since we had found some important resource differences between mathematics and other courses when we analyzed their costs above, we ran regression analyses both counting mathematics PAC courses and without counting such courses. We also explored alternative frames for analysis, indicated in a column titled "Comparison". Regression 1 compares 2v1,2, which means second (sophomore) year grades regressed on number of PAC courses taken in years 1 and 2, or in the freshman and sophomore years combined. The comparison indicator 3,4,v1,2 shown for regression 2 means the impact of PAC course taking in the freshman and sophomore years on grades in the junior and senior years, and so on.
Table 4: Effects of PAC Course Taking on Student Grades. Ref.# 
Group N Math?* Comparison** Criterion*** p(Signif.) b Beta 1 All students 336 N 2v1,2 NR 0.05 -0.02 -0.08 2 All students 336 N 3,4v1,2 NR 0.05 -0.02 -0.12 3 All students 336 N 3,4v1,2 PAB 0.05 3.32 0.11 4 All students 336 N 3,4v1,2 PC 0.10 -2.35 -0.10 5 All students 336 N 3,4v1,2 PCNR 0.05 -3.33 -0.12 6 All students 336 N 2v1,2 PAB 0.10 3.68 0.10 7 All students 336 N 1-4v1-4 PAB 0.10 2.37 0.09 8 All students 336 N 1-4v1-4 PCNR 0.10 -2.37 -0.09 
9 All students 336 Y 2v1 PAB 0.01 5.33 0.17 10 All students 336 Y 2v1 PCNR 0.05 -3.76 -0.12 11 All students 336 Y 3,4v1,2 PAB 0.10 2.57 0.10 12 All students 336 Y 2v1,2 PAB 0.05 3.26 0.11 13 All students 336 Y 3v1,2 PAB 0.10 3.52 0.10 14 All students 336 Y 1-4v1-4 NR 0.05 -0.49 -0.13 15 HS Rank 
90****184 N 3,4v1,2 PAB 0.05 5.04 0.18 16 HS Rank 90****184 N 1-4v1-4 PC 0.10 -3.58 0.17 17 HS Rank 90****184 N 1-4v1-4 NR 0.01 -0.66 -0.21 * Y = PAC course total including PAC math courses; N = PAC course total not including math
We used alternative criteria (percentage of As and Bs, percentage of Cs and NRs, and number of NRs) in our achievement model. Still referring to Table 4, "NR" indicates a regression predicting the total number of courses (in the outcome years designated) that students failed to satisfactorily complete or withdrew from (and were thus assigned NR or "no record" by the WPI Registrar). The term PAB indicates a regression that predicted the percentage of all of a student's grades that were As and Bs. PCNR means percentage of Cs and NRs out of total grades, and so on. (WPI's only grade options are A, B, C, and NR.)
** 2v1,2 = sophomore grades vs. number of PAC courses in freshman and sophomore years, and so on
*** NR = number of NRs (failure/withdrawal); PAB = % (A's + B's) total grades; PCNR = % (C's + NR's) total grades; and so on
**** High school class rank, percentile
Significance statistics displayed in Table 4 are standard significance levels for t-tests regarding the estimated PAC course coefficient values. These coefficients are shown as "b's". With reference to Regression 1 as an example, the "b" value is -0.02; this means that an added PAC course taken at the margin is associated on average with a 2 percentage point decrease in the number of NRs recorded for WPI students in the class of 1997. The Beta statistic shown in the right-hand column means that one standard deviation increase in PAC course taking by students in the class was associated with about 0.08 standard deviation decrease in number of NRs. Statisticians would call this an effect size of "0.08 sigma," or about one-tenth sigma in its order of magnitude.
The most substantial and important relationships with PAC course taking appear in regressions estimating the later effects of early PAC course- taking. This observation supports our initial hypotheses about PAC program effects. As shown in Table 4, these effects are the following:- Increased percentages of As and Bs in the sophomore year associate with increased PAC course taking in the freshman year (Regression 9).
- Increased percentages of As and Bs in the junior and senior years associate with increased PAC course taking in the freshman and sophomore years (Regressions 3, 11, and 15).
- Global Impact on Student Achievement
A general sense of PAC course impact stands out across the regressions summarized in Table 4. This is conveyed by the standardized or Beta estimates for most of the relationships shown. These lie in the vicinity of 0.12; the Beta coefficients for grade effects cited above are 0.11, 0.10, and 0.18 for regressions 3, 11, and 15 respectively. The beta coefficient for reduction in NRs is 0.12. This means that one standard deviation increase in PAC course taking is accompanied by approximately a tenth of a standard deviation in improvement in grades for the all-student sample, and approaching two-tenths sigma in one important analysis linking early PAC course taking with later course grades. - Other Effects In Our Models
Our regression models included indicators of "preparedness" as well as a variable showing number of PAC courses taken by each student in our databases. These indicators were percentile rank in high school class and mathematics SAT score. In general, Math SAT was not a significant predictor of any measure of academic performance, while high school rank in class was a significant predictor of some but not all grade outcomes (data not shown).
- PAC Impact on Grades
- Impact on Student Retention and Graduation Rates
We used our database to explore the possible influence of PAC course taking on student retention. For this analysis, we used a database overlapping the one we used for the analysis of PAC course taking on WPI student achievement. The main achievement data base included all graduates of the Class of 1997. For the retention analysis, we defined our data base by identifying all students who entered WPI in the fall of 1993. These students would become the graduating class of 1997 if they proceeded through their courses of study over 4 academic years, but some would leave and others fail to graduate on schedule for various reasons. Using this data base to probe effects of PAC course taking on retention and graduation, we selected the sub-sample of students who had obtained sufficient credits to have completed their freshman year with complete academic success (36 credits), thus ensuring they would have had opportunities to take a variety of PAC courses. We excluded from the analysis students who had transferred in more than one academic year's worth of credit (36 credits). We then divided this identified group into two outcome classifications for an analysis of retention effects. One group, the attrition group, eventually earned fewer than 100 total credits; the other group, the persisters, earned 100 credits or more. Since a WPI degree requires a minimum of 135 credits (WPI credits are not enumerated in the same way as those at other institutions), this dividing line approximates getting to near the end of the junior year at WPI. We then used a logistic regression to assess the impact of PAC course taking on student retention defined in this way. The regression employed controls for student high school rank and MSAT scores. The result was a PAC course taking effect on student persistence with a Beta weight of 0.17. That is, a standard deviation increase in PAC course taking (about 2 added PAC courses) was associated with about a one-sixth standard deviation increase in student retention in this model.
We used the same analytic structure for estimating PAC course taking effects on the probability that students entering in the fall of 1993 would graduate by May 1997. This logistic regression examined the effects of the number of PAC courses taken on whether students graduated on schedule or not. Control variables for high school rank and MSAT score were included, just as with the retention analysis. The result was a PAC course taking effect on student persistence with a Beta weight of 0.15. One standard deviation increase in PAC course taking (about 2 added courses) was associated with about one-sixth standard deviation increase in probability of graduation.
The analyses just described used data for the WPI class that entered in 1993 and expected to graduate in 1997, the class most fully engaged in the Davis initiative. We also explored effects of PAC course taking for the WPI class of 1996. The analyses for this class did not yield much in the way of significant relationships. This class had limited initial access to PAC courses beginning in the spring of their freshman year. Many PAC courses were not developed or operating in time for this class to have taken them.
Another way to understand PAC effects on retention and graduation is seen in Table 5, relating PAC course taking in the 1993 entering student cohort to the probability of eventually attaining 100 or more credits, and of graduating in four years.
Table 5: Relationship between PAC Course Taking and Retention/Graduation, Entering Cohort of 1993 # of PAC Courses Taken % of Students Retained % of Students Graduating N 0 72.2 69.0 126 1 91.8 88.2 220 2 95.8 86.7 143 >3 91.5 84.7 59
It is intriguing and noteworthy that taking one PAC course has a large positive effect on both retention and graduation, and additional PAC courses have no added value with respect to increased retention. One possible explanation is that one PAC course is sufficient for students to learn teamwork and communication skills that are of value in their later courses and projects. Another is that taking just one PAC course is sufficient for students to develop the social support network that is crucial to their persistence. It is important to note that most attrition occurs between the freshman and sophomore years, and among students who did NOT satisfactorily complete their freshman year (and who were excluded from this analysis). Additional analysis is required in order to identify whether particular PAC courses, or particular groups of students are primarily responsible for the results in Table 5. - Impact on IQP and MQP performance
- Project Grades
Since the project experience is an integral part of the WPI educational Plan, and since the skills acquired by students in the process of taking CL courses could reasonably be expected to constitute valuable preparation for project work, we wanted to know whether PAC course taking had any effect on project outcomes. For this analysis, we focused on the IQP and MQP, to the exclusion of the Sufficiency. The IQP and MQP are typically completed in the junior and senior years, beyond the time when a student is likely to have taken PAC courses, whereas the Sufficiency is typically done in the sophomore year. Also, the IQP and MQP are typically completed in groups, whereas the Sufficiency is exclusively a solo project, and group work is an important skill likely to be acquired in CL courses.
The independent variables were number of PAC courses, percentile rank in high school class, Math SAT score, and level of major difficulty (the latter for the senior survey responses only). Dependent variables were project grades and senior survey responses identified as relevant to IQP and MQP experience.
The analysis of effects on project grades was muddied by the fact that about 75% of MQPs and 82% of IQPs completed by the graduating class of 1997 were awarded a grade of A, and most of the rest received B's (Figure 2). This grade distribution is typical of other graduating classes; we chose to conduct the grade analysis on the data for the entering cohort (1993) corresponding to the graduating class (1997) in which overall grade effects of PAC course participation were most pronounced, as described above. With such a small grade spread, significant differences in performance would be very difficult to detect. Not surprisingly, no significant effects of PAC course participation on MQP or IQP grades were noted.
Figure 2: MQP and IQP Grades, Entering Cohort of 1993.
- Students' Perceptions of Project Experience
Senior survey responses for the graduating class of 1997 were analyzed for effects of PAC course taking on the students' perception of the project experience. The only significant effect observed was for MQP group harmony; notable expected effects that were lacking were students' perception of IQP group harmony, MQP and IQP group effectiveness, and MQP and IQP benefit to the students' education (Table 6).
Table 6: PAC Effects on Class of 97 Senior Survey Responses about Project Experience.
Significant results are in boldface.Senior Survey Question1 b (Slope) p (Signif.) Other Significant Predictors IQP benefit to your education (2b)2 -.065 .366 -- MQP benefit to your education (2c) -.015 .877 -- IQP group was effective (10) -.0039 .978 Major MQP group was effective (11) .078 .614 Major Got along with IQP group (12) .129 .286 -- Got along with MQP group (13) .307 .019 Major, HS Rank WPI's project-based system best (8) .135 .240 --
A broader view of the effects of the Davis program on project experience can be gained by comparing the responses of the senior class of 97, PLAs, and participating faculty with respect to the project experience (Table 7).
Table 7: Comparison of Senior Class of 97, PLA, and Faculty Responses to Project-Related Questions. Senior1/PLA2/faculty3 survey question Normalized4 mean senior response (Std. Dev.) Mean PLA response (Std. Dev.) Mean faculty response (Std. Dev.) Work in teams (5a/3a/1h) 5.8 (1.3) 6.2 (.88) 6.1 (.67) Solve open-ended problems (5c/3c/1o) 5.4 (1.4) 5.8 (.85) 5.5 (.93) Communicate effectively in writing (5e/3e/1g) 4.2 (1.7) 5.1 (1.4) 5.1 (1.4) Communicate effectively orally (5f/3f/1l) 4.6 (1.6) 5.6 (1.1) 5.8 (1.0) Find information on your/their own (5g/3g/1t) 5.6 (1.3) 6.0 (.96) 5.3 (.97) Use resources outside the classroom (5h/3h/1u) 5.4 (1.4) 6.0 (.93) 5.2 (.72) IQP group was effective (10,5) 5.2 (1.7) 5.2 (1.7) NA MQP group was effective (11,6) 5.1 (1.7) 5.6 (1.7) NA Got along with IQP group (12,7) 5.5 (1.7) 5.7 (1.3) NA Got along with MQP group (13,8) 5.6 (1.7) 5.1 (1.6) NA IQP was positive experience (14,9) 5.2 (1.9) 5.5 (1.3) NA MQP was positive experience (15,10) 5.2 (2.1) 5.5 (1.6) NA 1The full text of the senior survey questions can be found in Appendix B1. Senior survey questions numbered 5_ were preceded by the phrase, "Do you feel your education at WPI has prepared you well to:"
It is noteworthy that for most questions the PLA response was more positive than the senior class response. The two exceptions have to do with the MQP, which many PLAs had not yet completed, thus making the sample size for those questions very small indeed. It is also noteworthy that the senior class data showed much larger standard deviations than the PLA data, even though the sample size for the senior class was about six times as big as for the PLAs. This suggests a much wider range of opinion in the senior class than among the PLAs, and could be attributed to either the positive effect of the PLA experience, or the fact that PLAs were initially chosen from among our best students.
2The full text of the PLA survey questions can be found in Appendix B2. PLA survey questions numbered 3_ were prefffffed by the phrase, "How well has your education at WPI has prepared you to:"
3The full text of the PAC faculty survey questions can be found in Appendx B3. Faculty survey questions were preceded by the phrase "Compared to other classes you have taught, how do you feel the Davis approach to teaching impacted your students' ability to:"
4As originally administered, the senior survey offered respondents a range of 5 responses, where 3 was the neutral response. For this comparison, senior survey responses were normalized to a range of 7 responses, where 4 was the neutral response. The PLA and faculty surveys as originally administered offered respondents a range of 7 responses, where 4 was the neutral response.
The fact that faculty responses were higher (like those of the PLAs) with respect to teamwork, problem solving, and communication skills, but lower than those of both the PLAs and the seniors with respect to finding information and using resources outside the classroom, suggests that even with the PAC classroom model, other aspects of the WPI education (such as the IQP and MQP) are more valuable than coursework for imparting these skills to students.
- Project Grades
- Impact on PLAs
We also observed distinct Davis experience effects for PLAs -- effects brought about through serving as a PLA. These effects include: effectively manage one's time; communicate effectively orally; belief that working as a PLA positively affected ability to work in teams; function as a leader in a team. Figures 3-6 clearly show that serving as a PLA had very positive effects for those individuals. Faculty interviews confirmed this since several times faculty mentioned that perhaps the biggest beneficiaries of the Davis project were students who had the opportunity to be a PLA.
PLA training and job experience are unlike any other academic activity at WPI. As part of the "teaching team" in a PAC, PLAs learn and grow in ways they cannot in other learning contexts. The experience is so valuable that we come close to recommending that every student should serve as a PLA before graduating.
Figure 3: PLA Survey results regarding time management.
Figure 4: PLA survey results regarding oral communication skills.
Figure 5: PLA survey results regarding teamwork.
Figure 6
- Impact on Student Perceptions of WPI
Our senior survey analysis showed no significant correlations between Davis course taking and perceptions of WPI for the class of '96. The significant effects for the class of '97 are shown in Table 8. Students who took more PAC courses were more likely to claim a positive contribution from PLAs to their education, and indicated greater group harmony in their MQP. PAC course taking also contributed significantly to students' perceptions that their education prepared them to communicate well.
Independent of PAC course taking, the senior surveys from '96 and '97 showed some interesting results. Some of these are summarized in Table 9. The response key was a 5 = most important or superior, while a 1 = poor or no importance. The table indicates the mean response from a question. Our students rated WPI quite high in comparison with places like RPI, MIT, Clarkson, Carnegie Mellon, UMass, UNH, etc. Only one other school (Cal Tech) was rated at 4.07 and six other were ranked well below 4.0 (data not shown). Most students would recommend WPI to a friend (mean = 4.06), but were somewhat neutral about whether faculty cared about students and teaching (mean = 3.70).Table 8: Significant Effects of PAC Course Taking on 1997 Senior Survey Results b (slope) beta Signif. Level Other signif.
predictorsPLA contribution to my becoming
a successful professional.198 .179 .031 MSAT WPI education prepared me to
communicate in writing..179 .189 .035 MSAT WPI education prepared me to
communicate orally..227 .198 .020 -- MWP group got along well. .307 .196 .019 Major
The survey quite clearly showed that our students think the MQP was their most important academic experience, with large lecture classes being least important. Group work in courses and access to the technology were the next highest contributors. Consequently they felt that WPI prepared them well for working in teams. However, preparation for effective oral and written communication was not highly rated. Among all graduating seniors, TAs and PLAs were not ranked well as contributors to overall academic success (means less than 3.0) (data not shown).
Table 9: Mean Responses from Senior Survey, All Respondents. Independent of Davis Course Taking; Response Scale of 1 to 5. Senior Survey Question N 1996 1997 Quality of WPI's undergraduate program 8 4.17 4.12 Recommend WPI to a friend 46 4.08 4.04 Faculty caring about students and teaching 48 3.65 3.76 Benefit of MQP to education 11 4.72 4.63 Contribution of MQP to academic success 30 4.60 4.63 Contribution of project work to professional success 13 4.59 4.47 Contribution of newest technology to professional success 22 4.05 4.03 Contribution of group work in courses to professional success 19 4.03 3.89 Contribution of project work in courses to academic success 31 4.12 3.94 Preparation for teamwork 32 4.43 4.23 Preparation for effective written communication 36 3.46 3.12 Preparation for effective oral communication 37 3.50 3.40
- Continuation
The test of an externally funded innovation is the extent to which it continues after the funding expires. In that respect, the PAC model has enjoyed quite remarkable success (Table 10).- All of the participating faculty who have continued teaching the same course (10/12) have continued to use the PAC model.
- All 11 original PAC courses continue to be taught in the PAC format.
- All 7 PAC departments are supporting PLAs out of their department
budgets. In most cases, this reflects an increased budget allocation from
the WPI operating budget to support the use of PLAs, rather than a
reallocation of departmental resources.
- Cost Effectiveness Analysis
In this section of the report, we examine three issues: the costs of developing and offering PAC courses, the benefits (especially in terms of student achievement) associated with PAC teaching, and perspectives on relationships between costs and benefits. The main point of reference for the cost analysis is a comparison of the costs of PAC courses to the costs of traditional teaching designs. Our central focus for the benefits analysis is a quantitative analysis of student outcomes associated with enrolling in PAC courses.
Table 10: PLA Use1 by Department Biology Civil Eng. Chemical Eng. Computer Science Humanities Mechan. Eng. Math AY 92/93 WPI2 0 0 0 0 0 0 0 Davis3 7 0 0 0 0 0 0 AY 93/94 WPI 0 0 0 0 0 0 0 Davis 20 6 0 12 0 0 32 AY 94/95 WPI 23 10 0 2 0 0 0 Davis 0 0 6 0 0 8 56 AY 95/96 WPI 27 4 0 0 0 7 60 Davis 0 0 0 0 7 0 30 AY 96/97 WPI 21 43 0 0 17 0 50 Davis 0 20 4 3 0 0 39 Other4 0 2 0 0 0 8 0 AY 97/98 WPI 21 0 0 0 2 0 90 Davis 0 24 0 2 0 0 0 Other 0 0 8 1 0 0 0 Total by Dept. 119 103 18 20 26 23 357 1 Number of PLA-terms (1 PLA x 1 7-week term) per academic year.
Through five years of development and experimentation, participating faculty mounted a broad range of data collection activities aimed at documenting the nature of their redesigned courses and the impact of the program on student learners and other participants (for example, on PLAs and on faculty themselves). The analyses that follow draw on this outcomes data base as well as on a newly generated project-wide analysis of student outcomes from participation in PAC courses. Information was also generated regarding the resource requirements of various PAC and traditional course models through a specially designed survey of participating faculty.
2 PLAs paid from WPI funds. Some PLAs were used in courses that were restructured without Davis grant funds.
3 PLAs paid from Davis Foundation funds. Some were funded by Davis technology grants (see Appendix D).
4 PLAs paid from other (usually grant) funds.
- Course Reports Yielded Diverse Initial Data
There appeared to be considerable complexity confronting analyses of the effects of the WPI Davis project, generally reflecting the large number of frames that seemed potentially useful for weighing issues of educational productivity within this multi-year, multi-department project. The initiative attracted faculty from diverse disciplinary areas: biology; chemical, mechanical, and civil engineering; computer science; mathematics; and even one course in drama. Thus the project had potential impacts across areas where traditional models and norms of teaching would probably differ and where qualitatively different student outcomes would be measured in different ways. As the research team confronted the prospect of making sense of the project from a cost-effectiveness analysis standpoint, we had in hand a stack of project reports (prepared by faculty participants and summarized in Section V.A.) that described how each new course was designed and what each seemed to have accomplished. Understandably, the nature of the student outcome data drawn into these reports was highly variable. Introductory biology used one set of student self-report questionnaire items; another was created for introductory civil engineering. A mathematics course in differential equations repeatedly used the same student final examination and thus produced comparable measures of student learning on the domain of this exam, but similar comparative test data were not available in other disciplines. - Estimating Course Resource Demands
A cost assessment of an initiative such as the WPI Davis project demands information on both the costs of teaching in different ways and the costs of making the transition from one way of teaching to another - that is, initial implementation or start-up costs. The individual faculty course reports varied considerably in their attention to the development costs and potential or actual ongoing resource demands of individual courses -- both with respect to the new courses they had designed and the courses replaced by the PAC model. - An "Ingredients Approach" to Cost Accounting
We were interested in a full accounting of the elements required to teach in alternative ways and to accomplish the redesign of courses to achieve change. Levin (1985) describes this as the "ingredients approach" to cost accounting. We wished to document the full range of costs of the WPI Davis project, including faculty time to develop course plans initially; faculty time to plan subsequent versions of a PAC course; faculty time to train PLAs in the first edition of a course; training of PLAs in subsequent years by faculty or by experienced PLAs, perhaps at lower cost. Such development costs needed to be addressed for the simple reason that the true costs of adopting an alternative approach to teaching include the costs of planning and development to engage in the new design, as well as its ongoing maintenance costs.
We anticipated a need for some flexibility in how development costs would
be handled and reported for this work. We would begin by documenting
course development costs at the individual course level and would then
gain a sense of the distribution of development efforts and required
resources, resulting in a basis for assigning an appropriate share of
development costs to the ongoing offering of the new course designs. We
then amortized development costs over a suitable time horizon, effectively
adding an annualized share of course development costs to the operating
costs of a PAC course or sequence.
- Course Reports Yielded Diverse Initial Data
- Aggregation Issues
Our analysis also faced complexities associated with the aggregation of data. Starting with the individual faculty reports, we observed that most outcome data applied to specific classes-for example, student learning in linear algebra, or student self-reports about different types of learning in biology. It was also clear that some students had available to them sequences of PAC courses over their WPI careers, such as calculus, differential equations, and linear algebra PAC courses taken by many mathematics majors, or two specific courses within the biology department. The potential complexities are many: for example, results for biology students may reflect different patterns of PAC course participation within this specialty or major. Biology students may have taken one or more PAC classes, at different times, with different development and ongoing costs, and with different sets of outcome data available for analysis. Furthermore, students would have opportunities to cross departments and take PAC courses in both their major as well as allied sciences, computer science, and mathematics. Thus the project would be expected to have impacts on student careers at WPI that far transcend the impact of an individual course experience, or the impact of a within-department experience. - Devising a Data Collection and Assessment Matrix
We began with a thorough and systematic assessment of the information made available by the 11 individual faculty reports. We approached this by arraying key project data from these reports on outcomes, teaching models, and costs in a two-dimensional matrix -- in actuality a 14" by 17" spreadsheet with row-sections for each new PAC class and for all relevant comparison courses where data were available. The column-sections in this matrix reflected information on the key descriptive and evaluative dimensions of interest to our study. A student outcomes section contained descriptions of what measures were used, what results were reported, the nature of comparison measures for PAC versus "traditional" classes if any, and whether any results comparisons were reported in standardized terms. (Standardization was an important potential issue because we anticipated a need to aggregate individual course results that involved differing measures.) We also catalogued in this matrix information about the delivery models of courses described by the faculty -- how many of what sorts of sections and laboratories were offered each week, how many lectures, how many Teaching Assistant (TA)-led or PLA-led student classes, and so on. Another section of our matrix outlined any information about other costs of courses -- faculty planning time, training time, equipment purchases, numbers of PLAs employed.
Our initial thinking was that this class-level matrix would provide the building blocks for various subsequent levels of analysis. Potential examples included examining the impact of PAC courses on learning across several course offerings in a given major. Another was a search for more generalized outcome and cost patterns by examining results aggregated across different courses.
In fact, the resulting matrix accomplished an important representation of where things stood in appraising the WPI Davis project, but more importantly what would be needed to proceed with any sort of useful cost-effectiveness appraisal. The matrix proved an efficient way to visualize and appraise the results of 11 individual faculty studies. But as expected, the student results were reported in highly variable metrics, and through varying frames of analysis. The attention to cost data also proved highly inconsistent -- ranging from very little information that would support an appraisal of course delivery costs to informative portrayals of different course models and their impact on faculty time and other resource utilization. - Generating Additional Cost and Outcomes Data
The primary conclusion of the matrix-building exercise was that we would need to generate considerably more detailed course cost data in order to proceed with any cost-effectiveness appraisals of the WPI Davis project. We also saw that we would have to conceptualize PAC course-taking effects in a way that could be linked effectively to any cost data generated. These needs led us to two major data collection and analysis activities that dominated our work over 1996-97. The first was the creation and implementation of a course-ingredients survey. The other was an analysis of student transcript data obtained from the WPI Registrar, enabling us to identify the extent and nature of student participation in PAC courses and the relationships between PAC course-taking and important student outcomes. The discussion turns now to a description of these activities. - A Course Ingredients Inventory
We needed a thorough inventory of the costs of developing and offering PAC and comparison courses if we were to support any sort of cost-effectiveness analysis related to the WPI Davis project. As described above, the individual faculty course reports presented descriptions of the "ingredients" involved in traditional and PAC courses at varying levels of detail. Our new survey produf a review of each PAC course to construct as thorough as possible an inventory of what was involved in the development and delivery of these courses. This work also pursued an inventory of all ingredient costs of traditional courses to which the PAC courses should be compared.
The course ingredients survey was developed through discussions with the research team. A draft version was pilot-tested with two participating faculty before the full inquiry began. A copy of the final instrument is shown in Appendix B4. The survey was mailed to faculty who had taught traditional courses "replaced" by the new PAC model. Faculty were asked to complete the survey in time for an on-campus visit by Catterall. Faculty members presented their completed surveys in one-hour interviews which furnished them opportunities to seek clarifications or to provide supplementary details about their survey responses. The interviews also probed more general questions of faculty and other resource contributions to the experiment, and assessed their global impressions of the "costs" of teaching in their new models as opposed to traditional designs and the comparative benefits to students they perceived to be involved.
The survey data supported our general quest for understanding the impact of PAC learning on the costs of teaching -- especially allocations of faculty time for various purposes, the use of assistants, and the amount and types of resources required to plan and launch a new course. Its approach was to categorize and seek measures of resources needed for the most recent edition of a PAC course offering, and a suitable comparison course if available -- a course that intended to teach the same, or largely the same, content prior to the initiation of the PAC course. For most courses, there was an appropriate comparison course.
The course ingredients survey shown in Appendix B4 is divided into seven sections which help to classify the purposes of various resource commitments. The first section explores the resources devoted to the one-time initial development of the PAC and comparison courses. The second section asks about the resources required to prepare in advance of offering each type of course on an ongoing basis. Sections three and five list the resources required during a term to offer a course and explore this from different angles--the number of various activities such as faculty lectures and other sections attended by a typical student in a typical week, and the number of various types of events offered in a typical week to meet the needs of the course. Sections four and six account for paid support staff and other cash expenditures. Section seven tallies resources expended out-of-class, such as faculty office hours and student contact hours per PLA. - Results of Course Ingredients Survey
There are various ways that the results of this survey became useful to the research team. For the purposes of a cost-effectiveness assessment, perhaps the critical frame of reference is the marginal cost of providing the new model of teaching -- what resources did it take to launch, and what does it take to maintain the PAC models in comparison to traditional ways of teaching? Because the survey probed resource questions for a set of PAC versus comparison classes, we were able to generate general estimates of the differential resource requirements of offering PAC courses. This is a traditional marginal cost perspective -- the key question becomes what added (or saved) resources of what types were involved in the experiment. - Course Development Costs
In general, faculty reported investing about a full week of extra effort (an added 40.87 hours) to plan for the first offering of a PAC course, in comparison to what they experienced in the initial planning of the traditional comparison course (Table 11). They also used about 46 extra hours of student assistant time and a few hours of added clerical assistance. Table 11 also shows a trial amortization scheme for these development costs. Since the initial costs of developing a course typically provide benefits to multiple future offerings of the new course, we pro-rated the initial development costs over 8 future offerings, a figure which has since proven to be realistic. For example, the 40+ hours of added development time for a PAC course translate in Table 11 to about 5 extra faculty hours per term.
Table 11 goes on to describe other estimated differentials. Faculty in general claimed to put more hours into preparation for PAC courses -- particularly for PLA training. This training and other preparation time amount to about 12 added hours per term for faculty. - Cost Savings
Some resource savings for PAC courses begin to show up in the weekly format of courses. PAC courses typically involve one less faculty lecture per week and a small reduction of other faculty-led class sessions. But these small time savings are outweighed by reported increases in weekly class preparation time (an added 3.7 hours per week) and added office-hour time as well as added time interacting with students via e-mail. (While faculty reported additional e-mail with students as a part of the change to PAC, this phenomenon is probably part of a global shift to increased use of faculty-student e-mail in course delivery in higher education. We suspect that some of the reported increase should be attributed to general cultural developments and not to PAC.) The survey also accounts for the addition of paid PLAs to the PAC courses as well as levels of TA use between the two models, which show no differential. Also shown is the faculty estimate that on average, PAC courses did not involve differential cash outlays in comparison to traditional courses.
Table 12 shows cost extensions for the resource differential estimates shown in Table 11. Dollar estimates are provided for each ingredient. We used pro-rated average salary and benefit estimates for the key ingredient -- faculty time -- along with pay scales for other personnel including PLAs and TAs. Table 12 also contains a display that ignores the one time start up cost differentials. (Different decision making frames might benefit from this alternative accounting, even though it does not take full account of opportunity costs of the initiative.) As shown, the average net PAC course differential is in the vicinity of $5,000 in both accountings, and about $600 less if we do not attribute added e-mail time to the PAC model.
We also show cost differentials in Table 12 using an alternative valuation of faculty time. After polling a handful of individual faculty and the WPI Provost, we settled on a figure for the value of faculty time at consulting rates that is double the average salary/benefit rate for faculty (to $96 per hour from $48). This modification results in increased cost estimates for PAC, to more than $7000 (about $6150 with no student e-mail time differential).
The main result of our cost inquiry up to this point is that PAC courses on average are reported to be more expensive than traditional courses, by about $5000 to $7000 per term. At least half of this investment is in added faculty time and most of the additional half is cash outlays for paid PLAs. About half of that investment in faculty time consists of added time for class preparation during the term, usually to work with PLAs to plan small group work. - Mathematics Courses as Outlyers
In examining faculty responses to the course ingredients survey and in discussing survey responses with individual faculty members, it became clear that mathematics faculty as a group were, in comparison to faculty in other WPI departments, reporting rather larger time demands associated with their PAC courses. (It is not mathematics as a discipline that drives this increase in costs, but rather the model of PAC courses that WPI mathematics instructors chose.) We therefore calculated mean figures across cost categories for non-mathematics courses separately, to see what effects the mathematics data were having on the averages we showed in Tables 11 and 12. These new estimates are shown in Tables 13 and 14.
The non-math data are presented in a format paralleling the tables just discussed. There are several substantial differences when these data, reflecting only PAC courses in biology, civil engineering, chemical engineering, mechanical engineering, computer science, and dramatic arts (but no mathematics) are averaged. To begin with, the average differential resource contribution to initial PAC course development is much lower -- here an added 14.47 hours as opposed to more than 50 added faculty hours for all courses (the mathematics faculty reported an average differential in excess of 70 hours in this category). The non-math faculty also reported only half as much student assistance going into initial course preparation. Another substantial difference for the non-math group is the nearly three fewer hours per week of class preparation time reported -- this extends to nearly 20 fewer hours of faculty time, or half a week, per 7-week term. This group also reported a smaller differential in time devoted to keeping up with their students through e-mail and one fewer hour per week of office hours, as opposed to one additional hour reported when math courses were part of the group. - Evaluating These Time Differentials
As shown in Table 14, the estimates of cost differentials for non-math PAC courses tell a different story than the estimates for all courses taken together. The most important perspective generated is that the non-mathematics courses show an absolute decrease in faculty time devoted to a class over a seven-week term -- nearly 17 hours. The differential would be about 20 fewer faculty hours if we ignore the e-mail differential for reasons outlined above. Because there remains a cash investment in PLAs for these classes similar to that for all courses, there does remain a net investment required to maintain a PAC course. On average this net investment is in the $2000 range if faculty time is valued at a pro-rated salary figure. The cost differential diminishes to about $1000 when faculty time is valued at "consulting" rates and to as low as $777 when the reported e-mail interaction differential is ignored. Considering that the enrollment in these PAC courses is between 60 and 150 students, the cost differential per student in the PAC courses is relatively low, between about $5 and $35 per student for the entire course. - Summing Up the Cost Analysis
The analysis thus far attempts to estimate the costs of providing an alternative way to teach and organize introductory courses at WPI. The alternatives were created by participating faculty in response to small inducement grants, and the alternatives took on their own shape and character depending on individual faculty choices. There seemed to be a common approach reported in all five mathematics courses which caused these efforts to show up as comparatively resource-intensive in our accounting, including added up-front planning time and consistently more extensive work with PLAs during course terms. Accounting for and amortizing the initial investments faculty made in these course changes, and accounting for the "ingredients" that went into the regular routines of offering courses, we saw that on average, PAC courses required a net investment of an extra $5000 to $7000 per term. Taking the math courses out of the accounting produced estimates in the $1000 to $2000 range as the differential costs of providing a PAC course.
We intended this analysis to support a way of accounting for costs that would go quite beyond traditional and customarily casual ways of thinking about the costs of programs and initiatives in higher education. A common cost report would be the dollar amount of a grant that precipitates a program; another would be the "hard" dollar expenditures involved. But economists are quick to remind us that program decisions involve the use of resources that have value in alternative possible uses, and that this value is sacrificed when decisions are made to allocate resources in a particular way. This idea is called an opportunity cost perspective. In the case of the WPI Davis project, when faculty spend more or less time pursuing objectives as part of a teaching initiative, there are costs and savings to themselves and to the institution that should be estimated to achieve a true sense of the costs of a new program. This approach yielded the cost perspectives shown above. For the non-math faculty, the PAC course model does amount to an increase of faculty time available for alternative pursuits -- about half a week per 7 week term when teaching a PAC course. - Costs vs. Effects
The analyses presented above produce two sides of the WPI Davis project. First, a cost analysis based on an ingredients approach to cost accounting used a marginal cost perspective to compare teaching in the PAC versus traditional model. This analysis showed that WPI typically incurred net costs in the process of changing instruction under the initiative. When all courses were averaged together, the net costs were about $5,000 to $7,000 per term. When mathematics courses are taken out of the picture (5 of the 11 courses assessed), the net costs are much lower, about $1,000 to $2,000 per term for each of the remaining six courses.
The cost differential in non-mathematics courses is derived from two offsetting factors. The added cost is the approximately $2,400 per term paid to PLAs. A small saving in faculty time serves to partially offset the cash outlay for PLAs by somewhere between $400 and $1,400. So for these courses, a principal finding of the cost analysis is that faculty save some time which can be put to alternative pursuits -- research, other courses, committee work, or leisure. The net costs are largely devoted to hiring PLAs, who benefit individually from the experience as well as participating in an instructional design that brings benefits to other students. For mathematics courses, there is a considerably larger net cost associated with the PAC courses, accounted for by the costs of PLAs as well as added faculty time ranging in value from about $2,600 to $4,600. - Comparing Costs to Benefits
There is no pat way to link costs and benefits in this analysis, in part because the investment in PAC courses appears to produce benefits across a large spectrum. Improved upper division student grades, student retention, and student graduation rates are among the general effects as noted. In addition, our surveys showed positive impacts on PLAs, who benefit from the experience in terms of their own subject-matter knowledge, their academic self-concept, and in productive pursuit of their own career plans.
In this environment, linking a dollar investment in PAC to a percentage point increase in upper division grades -- a traditional bi-variate cost-effectiveness conception -- is a very limited way of expressing relationships between the costs and benefits of the initiative. Perhaps the best and most useful way to think about the results of our work is along the following lines. On the one hand, PAC involved patterns of investments in instructional resources approximating what we described. The principal findings were that PLAs cost some money, and for the non-math courses there was a somewhat offsetting saving in faculty time. For math courses, the expenditures for PLAs were equaled or exceeded by patterns of increased faculty time devoted to these courses. In return, we detected a substantial basket of outcomes that are significant. PAC courses had positive effects on student achievement, retention, and graduation. The project brought benefits to the many students tapped for work as PLAs.
In sum, there appear to us to be important effects that show the investments in PAC learning to be quite worthwhile. At the very least, the analysis will permit faculty, department heads, and the campus administrators to think through the implications of this initiative in considerable detail and texture, and to make future decisions about maintaining or expanding the initiative with more information than they might have obtained through individual faculty reports of effects and costs. This analysis may also provide a model for higher education institutions and state education agencies to use in determining the most productive allocation of increasingly scarce financial resources.
Finally, we hope that our efforts reported here will contribute to understanding of cost-effectiveness analysis in higher education instruction. Our work serves to underscore the complexity and multi-dimensionality of the enterprise. We also hope to illustrate the ingredients approach to cost accounting and how this produces quite different views of resource investments in programs than is typically revealed in program budget documents or grant agency ledgers. And there may be lessons in our use of the institutional student data base to model program effects: our approach using regression models with the WPI Registrar's data base seems more systematic and to account for more competing influences on effects, such as student background and department in our case, than are most efforts at program assessment in higher education.
There were additional outcomes brought about by the Davis project worth noting: 22 papers in professional journals, 30 professional presentations, 1 book, 4 externally funded grants, additional grants in preparation. The project has also spawned replications or off-shoots at MIT (TeamWorks in freshman mechanical engineering); University of Illinois, Champaign-Urbana (Engineering 100); University of New Hampshire, Durham (Math Department); and numerous professional inquiries. A complete listing of dissemination products is in Appendix C.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
* N = 12 PAC Courses and 9 Comparison Courses
** Net mean difference, PAC minus traditional courses
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
* Net mean difference, PAC minus traditional courses
** Consulting rate estimated at 2x salary rate
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
* N = 7 PAC Courses and 5 Comparison Courses
** Net mean difference, PAC minus traditional courses
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
* Net mean difference, PAC minus traditional course
** Consulting rate estimated at 2x salary rate
Last modified: Jun 21, 2005, 16:39 EDT
![[WPI]](/Buttons/seal.gif "WPI Homepage"){kind=small}
![[Home]](/Buttons/home.gif){kind=small}
![[Back]](/Buttons/back.gif){kind=small}
![[Top]](/Buttons/top.gif){kind=small}