Education Research

USE OF TEAM-BASED LEARNING IN GRADUATE EDUCATION IN THE BIOMEDICAL SCIENCES

Wayne T. McCormack and Cynthia W. Garvan, University of Florida, Gainesville, FL 32610 U.S.A.

Purpose  The focus of team-based learning (TBL) on problem-solving and integration of information has tremendous potential but has not been widely explored for improving biomedical science graduate education. This study assessed the TBL experience of biomedical science Ph.D. students and compared evaluations of courses taught with and without the TBL component.

Methods  Online course evaluation results were compared for prior to and following the implementation of TBL for two graduate courses. TBL sessions replaced nine of forty-two lectures in an immunology course. In a responsible conduct of research (RCR) course, TBL sessions replaced six small group case study discussions.

Results  After implementation of TBL, immunology students gave higher ratings for the course overall, general format (teaching methods used), fairness of test grading, appropriateness of test content, and overall rating of lectures. Student evaluations of the RCR course did not change significantly. A majority of students in both courses (62-75%) preferred TBL over other small group teaching methods. A majority of immunology students spent an hour or more preparing for class, agreed that they were better prepared and their classmates seemed better prepared for class, and over 80% agreed that TBL sessions helped them to be better prepared for exams. A majority of respondents from both courses agreed that TBL resulted in more interaction and that they could learn better in TBL compared to other small group settings in our curriculum.

Conclusion  Based on student feedback, TBL appears to improve course evaluations and to promote active learning in graduate-level courses.

Peer review of teaching study:  Designing, implementing, and evaluating a national faculty development program to peer review teaching based on an observation and feedback process.

Design group:  Elisa Zenni MD, Maryellen Gusic MD, Janet Hafler EdD, Wayne T. McCormack PhD, Carol F. Capello PhD, Maria Blanco EdD

Rationale:  Medical schools are seeing the need to better measure and reward faculty member’s educational contributions and clinical care performances, including the development of guidelines and valid and reliable comprehensive evaluation systems. Considerable research has addressed this issue (Jones & Froom, 1994; Lubitz, 1997; Jones & Gold, 1998; Green et al., 1998; Scheid et al., 2000; Snell et al., 2000; Nutter et al. 2000; Fincher et al., 2000; Bland et al., 2002; Williams et al., 2003). Researchers state that there is still a need for conducting and reporting on more teaching evaluation experiences that illustrate different teaching evaluation approaches (Snell et al., 2000; Simpson et al., 2004). Research also emphasizes that each teaching evaluation experience is unique to its particular context, and claims the need for new research to further these experiences (Rippey, 1981; Irby 1983a).

Peer evaluation of teaching can provide additional evidence of the quality of the teaching because a peer can give insights into instructors’ behavior in teaching settings; judge instructors’ instructional materials; evaluate instructors’ participation in curriculum development and educational research; and assess the quality of instructors’ academic, personal and advising skills (Irby, 1983a; Centra, 1993; Hativa, 2000; Berman, 2000; Colbeck, 2002). Although peer review is a well-established approach for evaluating research and patient care in academic medicine, it is less frequently used for evaluating teaching, and only a few medical schools have rigorous peer review of teaching (Irby, 1983; Mennin, 1999; Snell et al., 2000; McLean, 2001). Those who use peer review generally look at documents, for example, students’ evaluation, rather than using direct observation (Irby, 1983).

The purpose of the study is to design, implement and evaluate a national peer review of teaching program based on observations and feedback. The ultimate goal is to develop a program that can be adapted by schools. Included in the program will be tools that can be used across institutions to standardize effective teaching in the lecture and small group formats. Within the program, as we train faculty members to observe teaching and give feedback we will aim to develop and validate two instruments: one for lectures and one for small group teaching. We hope that this model will be generalized to any interested medical school.

Research question:  What is the impact of a peer review of teaching process? The study will specifically examine the impact on the reviewers and the faculty members who are observed and given feedback. The components of the study include a 90 minute faculty development workshop and the follow-up observations of teaching. This study will focus on the act of teaching and will not encompass all aspects of teaching but rather those amenable to being observed. Faculty teaching, which comprises instructional design and delivery skills; advising and mentoring; context expertise; teaching materials; and teaching leadership, is only one aspect of a faculty member’s educational contributions. We are hoping to support faculty who do not have access to receiving feedback on their teaching especially in non-lecture formats.

Time:  60 minutes
Place:  AAMC regional meetings
Faculty Presenter (SGEA):  Wayne T. McCormack, Ph.D.
Workshop objective:  To establish and standardize criteria for use in peer review of lecture and small group teaching
Expected Outcomes:  1. Refinement of tools developed by project subgroup;  2. Recruitment of a cadre of reviewers who are willing to use the tools in their own institution

Agenda:

A. Welcome and Introduction/Ice Breaker, 5 minutes.

B. Project Background and overview, 10 minutes. Literature review on use/impact of peer review (a handout) Review of national project a handout with bullet points and study design.

C. Small group work, 30 minutes. In the interactive session we will use two vignettes on a DVD as a stimulus discussion and use of the tools. In small groups each person will complete the assessment tools provided while observing the vignettes. Participants will be asked to take notes during the observation to support the scoring of the tool and also, take notes on their experience with using the tool. Discussion facilitated and notes taken by someone in the small group. What elements of the tools were easy to apply? And why What elements of the tools were difficult? And why Which elements of the tools resulted in the most and least inter-rater agreement? Are there items that should be added to the tools?

D. Large group discussion, 10 minutes. Summary discussion of the questions above to refine and enhance the tools. After all 4 workshops at each region have provided feedback the tools will be revised and distributed to the participants to complete their observations.

E. Conclusion, 5 minutes. Next steps for project/recruitment of faculty to participate in peer reviews.

Peer Nomination: A Tool for Identifying Medical Student Exemplars in Clinical Competence and Caring, Evaluated at Three Medical School

Wayne T. McCormack, PhD, Cathy Lazarus, MD, David Stern, MD, PhD, and Parker A. Small, Jr., MD

PURPOSE: Peer evaluation is underused in medical education. The goals of this study were to validate in a multi-institutional study a peer nomination form that identifies outstanding students in clinical competency and interpersonal skills, to test the hypothesis that with additional survey items humanism could be identified as a separate factor, and to find the simplest method of analysis. METHOD: In 2003, a 12-item peer nomination form was administered to junior or senior medical students at three institutions. Factor analysis was used to identify major latent variables and the items related to those characteristics. On the basis of those results, in 2004 a simpler, six-item form was developed and administered. Student rankings based on factor analysis and nomination counts were compared. RESULTS: Factor analysis of peer nomination data from both surveys identified three factors: clinical competence, caring, and community service. New survey items designed to address humanism are all weighted with interpersonal skills items; thus, the second major factor is characterized as caring. Rankings based on peer nomination results analyzed by either factor analysis or simply counting nominations distinguish at least the top 15% of students for each characteristic. CONCLUSIONS: Counting peer nominations using a simple, six-item form identifies medical student exemplars for three characteristics: clinical competence, caring, and community service. Factor analysis of peer nomination data did not identify humanism as a separate factor. Peer nomination rankings provide medical schools with a reliable tool to identify exemplars for recognition in medical student performance evaluations and selection for honors (e.g., Gold Humanism Honor Society).

USE OF TEAM-BASED LEARNING IN GRADUATE EDUCATION FOR PH.D. STUDENTS IN THE BIOMEDICAL SCIENCES

USE OF TEAM-BASED LEARNING IN TEACHING RESPONSIBLE CONDUCT OF RESEARCH TO FIRST-YEAR BIOMEDICAL SCIENCE GRADUATE STUDENTS

Aims and Objectives:  PhD students are required to take a responsible conduct of research (RCR) course, but existing courses were passive and lecture-driven. The main objective was to promote active learning so students could begin applying the principles of RCR using case scenarios and/or group problem-solving as they begin their dissertation research.

Methods/Approach:  The new course “Responsible Conduct of Biomedical Research” (GMS 7003) was established in Summer 2006 (http://idp.med.ufl.edu/RCR/). The course consisted of six Tuesday lectures followed by Thursday small group sessions, and utilized the instructional strategy known as team-based learning (TBL). TBL consists of pre-class preparation (e.g. lectures, reading), a readiness assurance process involving short individual and team tests, and the application of course concepts through problem-solving team assignments. Readiness assurance tests consisted of 10 multiple-choice questions based on assigned reading from "ORI Introduction to the Responsible Conduct of Research" by N. Steneck (2004). Problem-solving activities were developed with the lecturers.

Conclusions/Findings:  Student feedback about the TBL format was obtained via on-line course evaluations at the end of the semester. 68.5% of the respondents (54/65 students) preferred the TBL format over other small group formats used in other courses in our PhD program. 64.5% agreed or strongly agreed with a statement that they were better prepared for class, but only 18.9% thought their team members were better prepared (52.8% were neutral). 72.3% agreed or strongly agreed that the TBL format resulted in more interaction between students than in small group sessions in other courses, and 51.9% agreed or strongly agreed that they could learn better in team-based learning sessions than in small group sessions in other courses. Team-based learning appears to be a good format to promote active learning of RCR material. Next year’s course will include surveys of student attitudes, knowledge, and opinions about RCR issues before and after the course.

INITIATING A HUMANISM IN MEDICINE HONOR SOCIETY:  OPPORTUNITY AND CHALLENGE

Description of Issue and Rationale:

Since the medical professional is recognized not only by the technical competency acquired in his/her field, but also by the skills and attitudes exemplified in the personal interactions involved in the care of patients. medical educators must ensure that the educational and training environment strengthen the humanistic qualities that are essential to medical professionalism. One way to reward individuals who exemplify the highest standards of humanism and professionalism is through public recognition of excellence, such as the establishment of the new national honor society e.g. the Gold Humanism Honor Society (GHHS). The Arnold P. Gold Foundation is assisting medical schools in establishing local chapters. The process such schools use to identify and select students for membership must be credible, reliable and valid and is an area where there is much to learn.

The discussants for this small group session represent three medical schools that have initiated GHHS chapters at their respective institutions with shared and unique opportunities and challenges. Tulane University and the University of Florida have inaugural chapters and utilize peer nomination as the cornerstone of the selection process. Wake Forest University School of Medicine recently welcomed its first medical student members in 2004.

The session participants will consider the opportunities and challenges that sponsoring a chapter of the GHHS at their institution presents, the special circumstances that may exist, share in a discussion with others facing similar concerns, and learn from the experiences of the discussants.

Session Objectives:

Participants will:
   Learn about the national Gold Humanism in Medicine Honor Society initiative
   Consider issues and challenges involved in initiating or maintaining a local chapter of the GHHS at their school
   Identify specific strategies to utilize in initiating such a society at their home institution

Issue 1: Institutional Concerns
   What institutional benefits would there be to a humanism in medicine honor society?
   What institutional barriers would there be?
   How could those barriers be overcome?
Issue 2: Selection of Students:
   How would you select students for membership?
   How would students in the humanism honor society differ from those in AOA?
   What role should demonstrated community service play in the selection process?
Issue 3: Resident/Faculty/Community Members:
   Should residents, faculty, community physicians be included?
   How would they be selected?
   How could they participate?
Issue 4: Honor Society Activities:
   What would a humanism in medicine honor society do on a local level?
   What could it do on a national level?

Participants:

Discussant 1:  Cathy J. Lazarus, MD, FACP, Professor of Clinical Medicine, Director, Values in Medicine and Foundations in Medicine Programs.  Tulane University School of Medicine selected its first Gold Humanism in Medicine Honor Society student members in 2003. The Society has grown to include residents and faculty. The Chapter is now addressing issues related to sustainability and local and national visibility.

Discussant 2:  Wayne McCormack, PhD, Associate Dean for Graduate Education, University of Florida College of Medicine.  Dr. McCormack’s medical education research focuses on student peer assessment of professional competence, interpersonal skills, and humanism, and the possible predictive value of peer assessment throughout medical education and practice. Dr. McCormack is a faculty advisor of the UF Chapman Chapter of the GHHS, which inducted its first members in 2003. http://humanism.med.ufl.edu

Discussant 3:  T.J. Pulliam, M.D., FACG, Vice President for Professional Affairs, Assistant Professor of Gastroenterology, Wake Forest University School of Medicine.  Dr. Pulliam serves as faculty co-advisor for the newly formed WFUSM GHHS chapter which inducted its first students in the fall of 2004. A Leonard Tow Humanism in Medicine award winner, his interests include enhancing community and regional service to patients.

Abstract:  There is ongoing emphasis on professionalism and humanism in undergraduate and graduate medical education. One way to publicly recognize excellence in these areas is through the creation of a national honor society (The Gold Humanism in Medicine Honor Society). This session will address issues such as selection, induction, sustainability, and visibility involved in initiating a local chapter of the GHHS.

INITIATING A HUMANISM IN MEDICINE HONOR SOCIETY:  OPPORTUNITY AND CHALLENGE

Description of Topic and Rationale:  

      During the past decade, accrediting bodies, medical schools, individual faculty and others have made a substantial effort to focus on the teaching and assessment of "professionalism." While everyone agrees on the importance of this mission, the complexity of "teaching" the attributes considered integral to professionalism has proven to be a formidable task. Developing an effective assessment methodology within undergraduate and graduate medical experiences is an equally difficult job, yet one that both the LCME and the ACGME are requiring medical schools and residency training programs to do.
      Since the medical professional is recognized not only by the technical competency acquired in his/her field, but also by the skills and attitudes exemplified in the personal interactions involved in the care of patients, medical educators must ensure that the educational and training environment strengthen the humanistic qualities that are essential to medical professionalism. 
      It is a generally accepted principle in education that educators must inspect what they expect and reward what they expect to achieve the desired outcomes. One way to reward individuals who exemplify the highest standards of humanism and professionalism is through public recognition of excellence. One such mechanism is through the establishment of a national honor society promoting humanism in medicine, e.g. the Gold Humanism Honor Society (GHHS). With the support of the Arnold P. Gold Foundation, the GHHS is encouraging and assisting medical schools to create local humanism honor society chapters. The process such schools use to identify and select students for membership is the subject of ongoing scrutiny and discussion. Ensuring the process used is credible, reliable and valid is an exciting challenge and an area where medical schools and medical educators have much to learn.
      The moderator and discussants for this small group session represent four medical schools that have recently initiated GHHS chapters at their respective institutions with shared and unique opportunities and challenges. The session participants will have the opportunity to consider the special circumstances that may exist at their own institutions, to share in a discussion with others facing similar concerns, and to learn from the experiences of the discussants.

Questions for Discussion:

Group 1:  Institutional Concerns

1.      What institutional benefits would there be to a humanism in medicine honor society?

2.      What institutional barriers would there be?

3.      How could those barriers be overcome?

Group 2:  Selection of Students

1.      How would you select students for membership?

2.      How would students in the humanism honor society differ from those in AOA?

3.      What role should demonstrated community service play in the selection process?

Group 3:  Resident/Faculty/Community Members

1.      Should residents, faculty, community physicians be included?

2.      How would they be selected?

3.      How could they participate?

Group 4:  Honor Society Activities

1.      What would a humanism in medicine honor society do on a local level?

2.      What could it do on a national level?

Format:  

1.  The session will begin with a general introduction summarizing recent efforts to create a national humanism in medicine honor society (10 minutes).
2.  The participants will then be broken into smaller groups to address one of the four target areas, each facilitated by one of the discussants (35 minutes).
3.  The group will then come back together again to share the results of those discussions (30 minutes).
4.  The discussants will end by each sharing some of his or her school’s experiences (15 minutes).

USE OF PEER EVALUATION TO IDENTIFY EXEMPLARS OF HUMANISTIC BEHAVIOR

Wayne T. McCormack, C. Benjamin Stevens, and Parker A. Small, Jr., Dept. Pathology, Immunology & Lab. Med., Univ. Florida College of Medicine, Gainesville, FL
Cathy J. Lazarus, Dept. Internal Medicine, Tulane University School of Medicine, New Orleans, LA 70112
David Stern, Dept. Medicine, University of Michigan Medical School, Ann Arbor, MI 48109

Objective:  Peer evaluation is recognized as a valuable assessment tool to measure medical student performance in various aspects of professional behavior.  Summative peer evaluation data has been routinely collected for many years from University of Florida medical students at the end of their junior year, asking students to rank the top six classmates that best fit seven life situations related to clinical competence and interpersonal skills.  Such data provides a ranking that distinguishes 10-15% of the class for recognition of exemplary behavior in clinical competence and interpersonal skills.  The objective of this research is to determine whether this survey can be modified to identify exemplary behavior related to humanism in medicine, and to investigate relationships between rankings of these traits.

Methods:  Additional survey questions have been developed to focus on humanistic behavior traits related to patient care, and are being pilot tested at the UF and Tulane University.  Peer evaluation results will be analyzed using the SAS factor procedure and principal components analysis.  

Results:  Peer evaluation results and analysis from at least two medical schools will be presented.  Factor analysis will be used to determine whether questions related to humanistic traits identify a third factor that can be statistically separated from the clinical competence and interpersonal skills factors previously described using this method.  

Conclusions:  One way to foster caring in the practice of medicine is to recognize exemplars of such behavior to serve as role models.  Medical student peer evaluation may provide a powerful assessment tool to identify students worthy of special recognition for exemplary behavior related to humanism in medicine.  A peer evaluation survey that encompasses humanistic traits will be widely applicable to other schools, and may be used to recognize students who best demonstrate the time-honored tradition of caring in medicine.  (supported by the Arnold P. Gold Foundation)

Wayne and Terry Stratton (U. Kentucky) discussing the finer points of peer evaluation.
(Photo by George Samraj - thanks, George!)

CAN PEER ASSESSMENTS SERVE AS A TOOL TO PROVIDE CONSISTENT MEASURES OF HUMANISTIC QUALITIES?

This presentation is available for viewing as a PowerPoint file.

PEER ASSESSMENT IN UNDERGRADUATE MEDICAL EDUCATION:  AN INTERESTING CHALLENGE

Description of Topic and Rationale:  In light of the American Board of Internal Medicine’s (ABIM) decision to include peer evaluation in their re-certification process, the topic of medical student peer assessment is currently on the "front burner" in medical education.  This session’s overall goals are to generate and share new ideas by discussing innovative applications of peer assessment – as well as factors impacting its efficient use.  By examining processes used to conduct medical student peer assessment at selected different institutions, the small group discussion will focus on:  what information to collect, how and when to collect the information, how to use the information, and how the information can impact the professional development of medical students.  It is our hope that discussions of each of these areas will enable participants to plan and implement a systems approach to medical student peer assessment.

Questions for Discussion:

1.  What areas of medical student development are amenable to successful peer assessment strategies?
(a)  Are peer assessments best used in certain educational contexts (e.g. small group learning, laboratory group learning, case-based, problem-based learning)?
(b)  Can peer assessments be reliably and validly used to evaluate students’ skills in the areas of interviewing, physical examinations, counseling, teamwork, life-long learning, or professionalism?

2.  Should peer assessment be utilized throughout undergraduate medical school - or in only certain years or certain courses?

3.  What types of training/orientation, if any, should precede students’ assessments of colleagues?

4.  How should peer assessments be used and/or disseminated?  That is, should assessment results be included in students’ medical school records?  Should they be returned only in summary form to the individual student – or shared with residency programs through the Dean’s Letters?  Should they be used to drive curricular reform in institutions, and/or to remedy weaknesses identified by students’ peers?

5.  Do students trust the opinions of their peers in terms of professional development issues?  Can this feedback mold or change professional behavior of medical students?  Are there any latent (i.e. unintended) consequences to using peer assessments in undergraduate medical education?

Format:  We will begin with brief remarks by the moderator and discussants on the current peer assessment programs in place at different institutions.  The participants will then be divided into four groups with the moderator and discussants as facilitators.  The group discussion will focus on four areas:  What areas of medical student education are amenable to peer assessment?  What scales/forms/processes can be used in collecting information and do students require training to complete these assessments?  When is the best time to collect meaningful peer assessment information?  How the results could be used to maximize medical student development?  The discussion questions will be used to facilitate the small group discussions.  In the remainder of the session, the entire group will exchange information shared on each topic.  Ultimately, each participant will take back to their respective institutions ideas for possible implementation or refinement of peer assessment strategies.  In addition, the moderator and discussants hope to compile the ideas in this discussion group and subsequently, share them in a written format with the participants.

PEER EVALUATION AS A SELECTION TOOL FOR THE GOLD HUMANISM HONOR SOCIETY

Dr. McCormack, Associate Dean for Graduate Education, engaged in a project that brought him into contact with Dr. Parker Small, who instituted peer evaluations at the University of Florida many years ago. He was interested in finding a way to simplify the peer evaluation process used extensively by the University of Florida for many years. In addition, he sought to research possibilities for broader institutional uses from the results of this process. Of importance to The Gold Humanism Honor Society and for LCME purposes, he addressed what qualities might be measured using peer evaluations, including clinical competence, interpersonal skills and professional behaviors. He described to the group the difference between peer evaluations and peer nominations. He noted that peer evaluations ask students to rank every classmate in a series of questions or characteristics according to a fixed scale. The peer nomination process focuses on identifying those classmates who best fit a series of characteristics (e.g. who you would like to have at your side in an emergency).

At the end of the third year of medical school, University of Florida students were asked to rank the top six classmates that best fit seven life situations, four related to clinical competence and three related to social issues and interpersonal skills. Dr. McCormack noted that the questionnaire was a modification of one originally developed by Case Western Reserve. 

Summative peer evaluation results were used in Dean’s Letters. Each nominated student was assigned a point value, depending upon their placement on the ranked list of six students. A SAS factor procedure and principal components analysis was done and the score procedure enabled assigning each student a professional score and a social score. Dr. McCormack showed a factor analysis chart of peer evaluation questions for three graduating classes.

Dr. McCormack indicated that class ranking based on peer evaluation distinguishes the top 10-15% of the students, but does not differentiate among the rest. Using the professional and social scores, he noted that this identified two different subsets of students. He described earlier research by Dr. Small which demonstrated that peer evaluation of professional competence is best correlated with measures of cooperative learning and, to a lesser degree, with medical school GPA and NBME performance, and does not correlate with admissions data.

Dr. McCormack received a grant from The Arnold P. Gold Foundation to study the question: Can peer evaluation using factor analysis be simplified? He determined that the answer to this question was yes, and similar peer evaluation results can be obtained using factor analysis with three, rather than six nominations. He then asked whether a simpler spreadsheet analysis could be substituted for factor analysis. The answer to this question was also yes. He indicated to the group how he modified the variables:

• Number of students from 6 to 3
• Weighted variables points for placement to non-weighted (frequency of nominations)
• Factor analysis move to spreadsheet calculations
• Number of questions used
• Power of individual questions to establish the ranking

Dr. McCormack was able to show that the rankings based on spreadsheet analysis using frequencies of top 3 nominations distinguished a similar percentage of the class as factor analysis and that essentially the same top students emerge. He stated that although results using fewer than the original 7 questions are similar for the top 10 students, rankings begin to differ for the next 10-15 students. He suggested that the survey instrument might be simplified, especially if only trying to identify the top students.

In the grant from The Foundation, three additional items are being added to the survey: 

• Classmates who most demonstrate concern for patients, colleagues and the team;
• Classmates who have shown exceptional interest in service to special or under-served patient populations;
• Classmates you would want as a personal doctor for yourself or a loved one.

Dr. McCormack encouraged those in attendance to consider working with him in a collaborative project of Peer Evaluation as a Selection Tool for membership in the GHHS chapter. He indicated that this instrument also had possible longitudinal study options, comparing outcome measures in residency or practice. Several of the medical schools indicated an interest in working with him.

Implementing Medical Student Peer Evaluation Programs

Wayne T. McCormack
Dept. Pathology, Immunology & Lab. Med., Univ. Florida College of Medicine, Gainesville, FL

Peer evaluation is an assessment tool that is under-utilized in medical education, but may provide unique insights and reinforce other forms of evaluation. Medical student peer evaluation can be used to measure student achievement and predict future performance in such areas as clinical competence, interpersonal skills, and professional behavior. Having reviewed in a poster presentation entitled "Simplifying Medical Student Summative Peer Evaluation of Clinical Competence" our experiences and research at the U.F. College of Medicine, the objectives of this session are to provide a forum to discuss curriculum elements in use or planned at other medical schools that involve peer interactions and evaluation, and to discuss questions/concerns about implementing peer evaluation. Key discussion points will include the following:

1. The importance of providing peer interactions and small group teaching during the basic science years of medical education, e.g. Patient-Oriented Problem-Solving (POPS) system. Adequate peer interaction and teaching are required to facilitate valid peer evaluation.

2. The importance of introducing formative peer evaluation or feedback during the basic science years. Students should be given opportunities to learn how to evaluate their peers.

3. The use of summative peer evaluation in the clinical years, during which students have ample opportunities, e.g. clinical rotations, for the interactions that enable valid peer evaluations.

4. Comparison of methods of data collection and analysis. Participants are invited to bring copies of their survey forms for discussion and feedback.

5. The use of web-based on-line evaluations.

In concept, the use of peer evaluation has great face validity for identifying "good physicians" during undergraduate medical education, perhaps more so than grades and standardized exams. Peer evaluation is also an important factor for later professional success, e.g. referrals and hospital privileges, so it is important to provide our students with opportunities to learn and practice peer evaluation and feedback.

Wayne T. McCormack, C. Benjamin Stevens, Parker A. Small, Jr.
Dept. Pathology, Immunology & Lab. Med., Univ. Florida College of Medicine, Gainesville, FL

There is renewed interest in the use of peer evaluation to measure medical student achievement and predict future performance in terms of clinical competence, interpersonal skills, and professional behavior. Although a major determinant of later professional success, e.g. referrals and hospital privileges, peer evaluation is under-utilized in medical education. Summative peer evaluation data is routinely collected from University of Florida medical students at the end of their junior year, asking the students to rank the top six classmates that best fit seven life situations, four related to clinical competence and three related to social issues and interpersonal skills. Statistical analysis is used to establish two factors that are effective in distinguishing the top ~10% of the class. We have shown that peer evaluation of professional competence is best correlated with measures of cooperative learning and to a much lesser degree with medical school GPA’s and NBME performance, and is not correlated with admission data. To be widely accepted, peer evaluation ranking methods must be credible and relatively simple in data collection and analysis. We are using our peer evaluation data to identify the simplest, most reliable method of determining medical student peer rankings. Statistical methods are being compared, and the variables being tested include the number of students listed (six vs. three), use of ranked vs. non-ranked scores, number of questions used, and power of individual questions to establish the ranking. Preliminary results from three classes suggest that the survey instrument and data analysis can be simplified, because similar results are obtained using only three nominations and non-ranked scores. Interestingly, medical students appear to recognize their peers’ clinical competence and social skills as separate entities, which can be separated statistically, and that a "good doctor" must have both qualities.  (Supported by The Arnold P. Gold Foundation)