Davis Report - Methods

More Student Learning, Less Faculty Work? The WPI Davis Experiment in Educational Quality and Productivity

IV. METHODS

Course Initiatives

In each of the academic years 1992-93 through 1996-97, Requests for Proposals (Appendix A) were issued to faculty inviting them to apply for course redevelopment grants. These grants ranged from about $20,000 to $40,000, and typically covered summer salary for the faculty member, plus some combination of TA salary, PLA salary, and small amounts for supplies and travel. A committee consisting of the Davis Project PIs plus, in some cases, former PAC grantees reviewed proposals and made funding decisions. In some cases, revised proposal submissions were invited and ultimately funded. A separate WPI account was set up for each course initiative, with expenditures from that account made solely at the discretion of the faculty investigator. Faculty investigators were required to write final reports at the ends of their initiatives, which typically lasted for one course offering up to one academic year. In the case of the two pilot initiatives (BB1010/1020 and MA2051), a second request for funding for further course development was received and funded.
General Evaluation Methods
1. Objectives
  The goals of the Davis project evaluation program were to assess the project impact on:
  - Student learning within each disciplinary initiative and across the curriculum;
  - Student attitudes and satisfaction regarding their major, WPI, cooperative learning, project work, and teaching styles;
  - Student retention at WPI and progress toward graduation;
  - Students who served as Peer Learning Assistants;
  - Faculty involved in disciplinary initiatives regarding their time commitment and perceptions of student learning;
  - Overall costs and benefits for delivering a technical education using the PAC model.
2. General Methodology
  Our methodology used multiple measures that included a combination of qualitative and quantitative tools. An assessment matrix of ten key questions guided all evaluation design. These questions were:
  What was the effect of the Davis project on:
  - student performance in individual PAC courses?
  - student academic performance in junior and senior years?
  - graduation and retention rates?
  - student performance in IQPs and MQPs?
  - students' teamwork skills in IQPs and MQPs?
  - students who served as PLAs?
  - student perceptions of their overall experience at WPI?
  - faculty continuance of Davis-funded initiatives?
  - the cost effectiveness of teaching with a peer-assisted CL structure?
  - the cost effectiveness of the total project?
  Table 2 presents a summary of the types of effects studied and the data sources used. Except for the retention analysis, multiple data sources were used in each case. Each individual initiative used assessment tools specific to the discipline and course, but most employed a student survey. The Davis project team read final reports from all disciplinary initiatives and compared results from each for common themes.
  
  We assessed the impact of number of PAC courses taken on student grades in upper level projects and courses using the Registrar's student record database. Students within PAC courses, graduating seniors, PLAs, faculty advisors of MQPs and IQPs, and faculty funded by the Davis project were surveyed. Our consultant interviewed all faculty involved in Davis-funded initiatives. He also designed and conducted a cost-effectiveness analysis of each disciplinary initiative. Details of all these tools and specific methodologies are provided below.
Specific Assessment Methodologies
1. Educational Outcomes
  Learning outcomes in individual initiatives were measured by a variety of tools depending upon the course structure. Typical measurements included student grades on exams, group project reports, group oral presentations, and homework. In many cases it was possible to compare PAC-taught students to a comparison class of non-PAC students. The comparison class was a "traditionally" taught section (for example, one that primarily employed lectures). Usually the comparison course was taught by the same professor involved in the PAC course, and sometimes by a different professor. In all comparison studies, every effort was made to minimize differences between comparison and PAC courses. It was then possible to compare the learning of the same course material between PAC and traditional students. The details of each disciplinary assessment plan are found in the final reports of each project. The primary question answered in each project was "What was the educational impact on students within individual courses taught in the PAC format?"
  
  Table 2 unavailable
  Our general model for PAC effects had additional characteristics. Our theoretical reasoning was that PAC course taking in the early going -- the freshman and sophomore years -- might pay off in increased success in the junior and senior years, as well as in increased retention and graduation likelihood. Our quest for general relationships between PAC course taking and student outcomes was carried out by extensive analyses of data on two recent class cohorts at WPI. We examined data for the Class of 1996 (those students who graduated from WPI in May 1996) -- the first class that had appreciable opportunities to take PAC courses beginning with the inaugural PAC effort in the biology department in the spring of 1993. We also examined data from the Class of 1997 whose students graduated in May 1997 and who had substantial opportunities to both take PAC courses and to demonstrate various effects including their propensity to graduate.
  
  The student data base generated by the WPI Registrar for this project contained various information important to our analyses. We relied on the following statistics for our work:
  1. Student grades by course by year.
  2. Numbers of PAC courses taken by year.
  3. Student major department.
  4. Student math SAT score.
  5. Student rank in high school class.
  6. Student credit hours earned by year.
  7. Student graduation status as of May 199-- yes or no.
  We analyzed how PAC course taking in the first two years at WPI affected student performance in upper level coursework. Percentage of As, Bs, and Cs, and number of NRs, in junior and senior level courses were regressed against the number of PAC courses a student took in the freshman and sophomore years. We controlled for high school class rank, math SAT score, and major discipline. We approached an analysis of the impact of PAC course-taking on student grades by estimating the following linear regression equation:
  EQ. 1: Grades = b1*PAC + b2*MSAT + b3*HSR + b4*DEPT + e
  where PAC is the number of PAC courses taken by an individual student; MSAT is Math SAT score; HSR is percentile rank in high school class; DEPT is a major department code; b1, b2, b3, and b4 are regression coefficients; and e is an arbitrary constant. The general logic of Equation 1 is that a student's grades might be influenced by the extent of their experience in PAC courses, but that such a relationship, if found, should enlist controls for student background and for the department they majored in. We reasoned that grades may be systematically influenced by student mathematics ability or general achievement level upon matriculation to WPI. Thus the equation includes variables indicating Mathematics SAT score (MSAT) and percentile rank in high school class (HSR). We also observed in our initial exploration of the WPI student data base that the average grades of students varied rather sharply by department. The percentages of As, for example, among all course grades on student transcripts ranged from about 20 percent in some departments to about 40 percent in others, in a fairly even spread. Based on the "grade difficulty" criterion, we assigned a numerical value between one and three to each major department. At the same time, students' tendencies to take multiple PAC courses also varied somewhat by department, since some departments offered more PAC course opportunities than others. We say "somewhat" on this count because students were able to take PAC courses both within and outside their departments -- students in engineering majors taking PAC editions of mathematics courses, for example.
  
  Our statistical analyses were carried out using data bases developed by the WPI Registrar. We examined data separately for the classes of 1996 and 1997 and did not include transfer students. The general procedure for estimating the coefficients in the regression equations was the Ordinary Least Squares procedure, a robust test for independent associations between predictor (or independent) variables and an outcome (or dependent) variable. The resulting coefficients are estimates of linear slope coefficients describing the relationship between changes in each predictor variable (for example, the effect of an added PAC course during the freshman and sophomore years) and the change in the outcome variable (for example percentage of As and Bs in the junior and senior years).
  
  The regression coefficients in Equation 1 have three important characteristics in this type of analysis. One is their sign -- whether the relationship is positive or negative. Another is their absolute magnitude -- how much change in an outcome variable is associated across the data base with a one unit increase in a predictor variable. In general, we estimated the effects of an added PAC course for our analyses. And third, the regression procedure provides a test of significance using the t-distribution for estimates of errors around reported regression coefficients. The result is the common reporting of the probability that the true value of a regression coefficient is actually zero; standard practice in statistical inference is that those statistics for which the probability of a zero value is less than .05 are considered statistically significant.
  
  The regression procedure also estimates the relationships between predictor and outcome variables in standardized form. This is the Beta statistic which describes the change in an outcome variable in standard deviation units associated with a one standard deviation change in predictor variable. This statistic is useful in comparing the relative size of effects when different variables using different metrics are used in an analysis, for example comparing the relative effects of SAT scores and PAC course-taking.
  
  We also explored the impact of PAC course taking on two additional student outcomes considered important to institutions of higher education, the propensity of students to remain enrolled longer at WPI (retention), and the probability that students would eventually graduate.
  
  Our analyses of retention and graduation effects used the following procedure. We created a data base of all students entering in fall of 1993 -- those students who would be most likely to graduate in spring of 1997 by completing their WPI courses of study over four academic years. We then identified all students in the data base who had completed 36 credits, representing the accomplishment of successful completion of the freshman year. We then divided this group: for the retention analysis, we compared those among this group who completed 100 or more credits against those who completed fewer credits (between 37 and 99 credits) by fall of 1997. We estimated the impact of PAC course taking on the probability of earning 100 or more credits as an estimate of retention effect. Analogously, we divided the same group into two outcome groups for the graduation analysis: those who had graduated by fall 1997 and those who had not. We used regression analysis as a robust estimator of the effects of PAC course taking and concentrated on the significance of predictors and Beta effects. There is no straightforward interpretation of the specific regression coefficients since the dependent variable is binary. We also show classification tables indicating percentages of students persisting and graduating by PAC course taking levels.
  
  These three analyses (capturing relationships between taking PAC courses and student achievement, retention, and graduation) were carried out under a common theoretical orientation. This was grounded in the idea that PAC courses should have positive developmental and learning effects on students. More precisely, we reasoned that the newly organized courses emphasizing problem-focused work and cooperative learning would leave students better equipped for advanced courses that required skill at problem solving and group work. We also hypothesized that both this added type of success and effective preparation along with the interpersonal experiences of group work in the early years at WPI would socialize students in ways that might prompt increased student retention (remaining enrolled) and higher likelihood of graduating.
2. Student Course Evaluations
  The standard WPI course evaluation summaries were analyzed for all individual initiatives. In cases where comparison courses were available we compared student responses for the PAC version to the traditional version. We looked for statistically significant differences for each question between PAC course responses and those for the comparison course.
3. Senior Survey
  We surveyed the graduating classes of 1996 and 1997. An incentive for survey completion was a donation to the class gift fund in proportion to the number of surveys returned. The survey is included here in Appendix B1. Each member of the senior class was solicited by email to complete an electronic or paper survey. The survey queried student attitudes, experiences, and satisfaction with their academic career, their perceptions about learning, and their preparation for a career. We received completed surveys from 231 students from the class of '96 (44% return rate), and 172 from the class of '97 (32% return rate).
  
  We analyzed survey results by looking for differences between students that took no PAC courses while at WPI and those that took between one and five PAC courses. Regressions of individual question responses on PAC course number were considered significant if the calculated significance level was less than 0.05.
4. PLA Survey
  Surveys were mailed to all students who served as PLAs between 1993 and 1996. The survey is attached in Appendix B2. Questions were designed to understand how the PLA experience affected each student's WPI academic experience and their preparation for a career. The survey included a section for written responses to specific open-ended questions. These responses were analyzed qualitatively for common elements. We mailed 102 surveys and 38 were completed for a return rate of 37%.
5. Faculty Survey
  Each faculty member who conducted a PAC course initiative (12 total) completed a survey designed to query their attitudes about the PLA model, general teaching and learning styles, and their perceptions of student learning related to teaching method. Open-ended responses to specific questions were also included. A copy of the survey is in Appendix B3.
6. Course Ingredients Survey
  Each faculty involved in a Davis project completed a survey designed to understand the costs associated with the PLA model. A copy is shown in Appendix B4. Twelve surveys were completed. Faculty were asked to compare things like course preparation time, course resource costs, and ongoing course maintenance time between their traditional version of the course and the PLA-based version. Catterall analyzed the results of these surveys and integrated them with his interviews (below) to draw conclusions about costs of PAC course delivery.
7. Interviews
  All twelve faculty who completed the cost-benefit survey were interviewed by Catterall. These interviews helped us understand how faculty arrived at the numbers in their cost-benefit survey. Open-ended responses to specific questions were also recorded. Qualitative interview responses were integrated with the faculty surveys and cost surveys to obtain a sense of how faculty who taught using the PLA model liked it.
8. Summary
  Figure 1 illustrates the relationship among the various assessment tools used in the project. We describe the detailed results from all evaluation measurements below.
  
  Figure 1: Overview of the Assessment Plan for The Davis Project

Maintained by webmaster@wpi.edu
Last modified: Jun 21, 2005, 16:42 EDT