SPECIAL: OFFICE-BASED MEDICAL STUDENT TEACHING

Effective Feedback and Evaluation in Clinical Medicine

Michael Brousseau, M.D.

For the past ten years, internal medicine has been moving from the inpatient to the ambulatory setting for the education of both medical students and residents, prodded for the most part by the need to better prepare graduates for the ways in which medicine is now practiced. Accrediting agencies and board examinations have contributed to this realignment of educational experiences. At UCLA, the Primary Care Network has played an important role in meeting the increasing demand for ambulatory-based education in internal medicine. For the most part, studies of medical education have demonstrated equivalent learning outcomes between inpatient and ambulatory rotations.¹ However, in a large review of studies of ambulatory education, Irby suggested that the provision of direct supervision, accurate evaluation, and frequent feedback remain critical issues for education in the ambulatory setting.²

Although preceptors enjoy the teaching interactions with students in the ambulatory setting, many do not feel at ease with evaluating clinical performance or providing feedback. Most spend only a limited time with a single student, and therefore, find it difficult to distinguish between an advanced medical student and an intermediate student, or an intermediate student and a poor student. Unable to provide a detailed evaluation of a learner's performance, physicians have too commonly ascribed to the adage of "I know a good student when I see one." One-half of UCLA students graduating in 2000 indicated on a survey at the end of their third year that they had not been directly observed while conducting a focused history or physical exam as part of the Doctoring 3 preceptorship. Part of the difficulty in providing feedback may be that the data needed to evaluate performance are not being collected.

Unfortunately, there is no single assessment method that can be used to evaluate clinical competency. The Accreditation Committee for Graduate Medical Education has recently published an on-line compendium of evaluation tools (http://www.acgme.org/outcome/toolbox.asp) and is increasing the pressure on residency programs to use a variety of these tools to document the quality of resident performance. In a review of measures of clinical competency, Holmboe and Hawkins demonstrated that a combination of assessment tools provides the best opportunity to evaluate physicians in training.³ The authors noted that the patient-based, clinical evaluation exercise of the American Board of Internal Medicine (Mini-CEX) is very good in assessing the skills of history taking and physical examination. The Mini-CEX is only moderately effective for assessing communication and humanistic skills and inappropriate for judging the resident's knowledge base, clinical judgment, and procedural skills.

Rating scales are the most common form of evaluation used by preceptors. Noel et al demonstrated that attendings could more accurately assess resident performance using a tool with clear "anchors" for each point on the scale that prompted the respondent what particular ratings would mean for the dimension being evaluated.⁴ The study design randomized 203 faculty internists to one of three groups using either an open-ended evaluation form or a structured form that included explicit descriptors for each level of performance. A third group used the detailed evaluation form after seeing a videotape showing good evaluation techniques. The results showed that when observations were not prompted, participating attendings recorded only 30% of a resident's strengths and weaknesses based on a videotape of his interview and examination of a single patient. Among participants using the structured form, accuracy improved to 60%. The videotape group showed no improvement over the non-videotape group.

Difficulties in assessing clinical competency for purposes of evaluation are also reflected in preceptors' failure to provide adequate feedback to learners. An annual survey of graduating medical students conducted by the Association of American Medical Colleges demonstrates that medical students are not provided with sufficient feedback and evaluation, both nationally, and at UCLA.⁵ A study by Hewson, et al at one large academic medical center showed that only a minority of residents received any type of feedback during their clinical rotations.⁶ Only 40% of the residents reported receiving regular written evaluations. Fifty percent reported receiving verbal feedback and most of this was positive. Less than 10% reported any corrective feedback.

This article explores the barriers faced by preceptors in conducting evaluations and providing feedback to learners in the ambulatory setting, suggests strategies for use in providing feedback throughout a rotation, and presents a simple framework for evaluating clinical performance and focusing feedback.

Feedback Versus Evaluation

To utilize effective feedback and evaluation, the preceptor must understand the subtle differences between these two concepts. Feedback and evaluation are linked but different.⁷

Feedback is a tool that is formative in nature. Feedback has the quality of coaching and is an essential component of professional growth and development. Feedback with students should occur with each patient encounter or at the end of each clinic session.

Reinforcing feedback emphasizes the correct actions of the learner. An example is "I was impressed that you gave Ms. Jones plenty of time to tell her story." Corrective feedback encourages changes in behavior. An example of corrective feedback is "When you palpate the liver it is important to disrobe the patient."

Evaluation, on the other hand, is summative. Upon completion of a period of time together, the preceptor must determine whether the learner has met the objectives of the course or rotation. Evaluation is judgmental and can take place at the midpoint or end of a course. It provides quality assurance for the medical profession and a means of documenting students' accomplishments.

Overcoming Barriers to Feedback and Evaluation

Although the concepts of feedback and evaluation are simple, many barriers keep preceptors from comfortably utilizing these tools. Commonly, preceptors are unsure of the reliability and validity of their evaluations. They do not know what students and residents should be expected to master at their respective levels of learning. Furthermore, preceptors do not feel that without more extensive contact they cannot fairly assess a student's or resident's level of achievement. Preceptors are also worried about the effects of an evaluation upon the learner's career and emotions.⁸ Although physicians feel comfortable in the discussion of a new diagnosis of cancer with their patients, they are insecure about an upset student and do not want to impair a promising career with a critical evaluation. Two years ago, an "off the record" section was added to the medical student clinical rating form to encourage faculty members to share their concerns in such a way as to not affect the actual record of that student's performance.

A careful way of overcoming these barriers is to announce feedback and evaluation intentions at the start of any rotation. The teacher should discuss with the learner (1) the course objectives, (2) a schedule of when feedback and evaluation will occur, and (3) strategies that will be utilized to assess progress toward those objectives. These points are discussed in greater detail in the first article in this series, Orientation and Priming: Setting the Stage for Learning.

The objectives for an ambulatory medicine rotation as outlined by the American Board of Internal Medicine for residents can be helpful in setting objectives for medical students as well. Students should be assessed on their mastery of three broad areas. First,their fund of knowledge should be judged in light of its extent, organization, and up-to-date nature. Clinical judgment and quality of their medical care should be assessed in light of the learner's ability to apply that knowledge base to the patients seen. Second, learners' attitudes are measured through their exhibited qualities of professionalism and humanism as well as their ability to communicate with patients, families, and other members of the health care team. Finally, skills such as oral case presentations, histories, and physical examinations should be observed and evaluated.

It is important to set up and follow a schedule of feedback and evaluation sessions. Students need to be told that regular feedback will be given with the intention of making them better physicians. Feedback can be provided after every patient or at the end of each clinical session, depending upon time constraints. A midterm and final evaluation can be scheduled according to student needs and course requirements. Sessions can be one-on-one or performed in a small group, depending upon the skills and sensitivities of the learners.

Strategies for assessment and measurement should be discussed with the learner at the beginning of the rotation. Preceptors typically use three approaches in collecting data about student or resident performance. Case presentations allow the preceptor to determine how well the learner can synthesize the data collected during the patient encounter based on how well the learner diagnosed the patient. This approach is limited to inferences about the accuracy and effectiveness of the skills used by the learner in collecting the data. Direct observation by the preceptor is needed to determine how well the learner is performing the history and physical exam, completing technical procedures, or educating patients on their medical problems. The third most common method of assessment is chart review. Learners should be informed about which of these the preceptor will use in a particular rotation or course.

For residents, preceptors must use the American Board of Internal Medicine Evaluation Form with ten domains and a nine-point scale. Residents are also tested with an annual written in-service training examination and via an observed work-up. Preceptors in Doctoring and Fundamentals of Clinical Medicine will need to evaluate students on the standard rating scale or checklist provided by the School. Third-year medical students in the Ambulatory Medicine Clerkship are evaluated on a seven-point scale for eight competencies. A major disadvantage to the use of rating scales is a "halo" effect, with an overwhelming majority of residents or students receiving scores at the top of the range, even when other indicators suggest poor performance. An analysis of internal medicine rating forms for medical student on the core clerkship in 1999-2000 revealed that only one student was rated on any dimension below a "4" - a point on the scale which indicates a satisfactory performance.

RIME

In seeking to improve the use of rating scales and the narrative descriptors often required on such forms, Pangaro has developed an innovative method of describing the progress of trainees from "reporter" to "interpreter" to "manager" and "educator" (RIME).⁹ The framework emphasizes the progress over time, and distinguishes between basic and advanced levels of performance for both ward and clinical rotations. The implementation of this method has provided several advantages. Multiple raters describing the same students have reached a high level of consistency with reliability of 0.8. The RIME system is also highly correlated with how an individual student performs in other rotations. RIME is easily applied and has increased the number of observations being collected on each student. The prime virtue of the RIME terminology is its portability and ease of use by all teachers.

Reporter

At the reporter level, the learner accurately gathers and communicates clinical facts about his or her patients. The reporter has mastered basic clinical skills and can apply knowledge of what is normal and abnormal in selecting what should be reported. This learner can identify and label new problems and demonstrates a consistency of clinical skills. The student should demonstrate a sense of responsibility to the patient by participating fully in clinical rounds and following up on all assigned responsibilities. These skills are generally mastered during the third and fourth year of medical school.

Interpreter

The transition to interpreter is often a difficult one for students. As an interpreter, the learner must prioritize among problems that he or she has identified for a particular patient. The interpreter is able to provide a differential diagnosis with at least three possibilities, each supported with history, physical exam and laboratory findings. These skills are generally mastered by the end of internship by which time the learner can consistently interpret common medical problems accurately.

Manager

The manager stage requires more confidence, knowledge, and judgment. A manager decides when action should be taken. He or she proposes and selects among diagnostic and therapeutic options for the patient. A manager is able to take into account the patient's particular preferences and circumstances. For example, the manager can integrate the patient's preferences and beliefs in making decisions concerning end of life care or prescribing medications. These skills are generally mastered during residency.

Educator

The educator demonstrates a thorough mastery of basic clinical skills and accuracy in diagnosis. The educator, however, goes beyond the known. He or she reads about the patients seen, and beyond, and shares new learning with others. Educators define important questions to research. This level of clinical practice requires more drive, insight, skills, maturity, and confidence. Not all trainees reach this stage.

The instructor should keep in mind that a learner must show consistency at each particular stage before being labeled as having advanced to the next level. It is not unusual for a student to be a novice on a complex case and close to a manager in a simpler and more common case.

Feedback Progress Notes

A simple method of tracking the learner's competency during patient encounters is through the use of feedback progress notes. A feedback note allows documentation of the clinical performance on a case-by-case basis. Multiple progress notes can be used to identify patterns of strengths and weaknesses across multiple encounters and provides a record for more specific formative and summative evaluations. Each note details the stage of progress of the learner using the RIME system and prompts the preceptor to provide positive and constructive feedback. A feedback progress note contains the following information: student name, patient name, date, RIME level, does well, and next step. Appendix A includes examples of three case presentations summarized by the preceptor in the accompanying feedback progress note. The note is intended to be shared with the student.

Conclusion

Although the process of feedback and evaluation is often avoided for emotional reasons, it can be accomplished with a minimum of pain by using the RIME system to provide effective and time efficient feedback. The evaluator should craft the "next step" for the learner based on the degree of mastery of the current RIME level. The feedback progress note provides a means of documenting clinical performance with each patient. Notes can be stored for discussion across events or shared with the learner at the end of each encounter as suggested in the "One Minute Preceptor" described in one of the articles in the teaching series.

Evaluation and feedback should be based on case presentations, direct observation, and chart review. The student should be allowed to perform the majority of the work with selected patients and expected to grow their knowledge base through self-directed study stimulated by the patient encounter. More responsibility needs to be accompanied by more feedback. Give the student a chance at a high pass.

REFERENCES

1. Pangaro L, Gibson K, Russell W, Lucas C, Marple R. A prospective, randomized trial of a six-week ambulatory medicine rotation. Acad Med. 1995 Jun;70(6):537-541.

2. Irby DM. Teaching and learning in ambulatory care settings: a thematic review of the literature. Acad Med. 1995 Oct;70(10):898-931.

3. Holmboe ES, Hawkins RE. Methods for evaluating the clinical competence of residents in internal medicine: a review. Ann Intern Med. 1998 Jul 1;129(1):42-48.

4. Noel GL, Herbers JE Jr, Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do internal medicine faculty members evaluate the clinical skills of residents? Ann Intern Med. 1992 Nov 1;117(9):757-765.

5. Association of American Medical Colleges 2000 Medical School Graduation Questionnaire. Washington, DC: AAMC. 2000.

6. Hewson MG, Fishleder AJ, Halperin AK, et al. Educating residents for managed care: report on a multidisciplinary conference. Acad Med. 1998 May;73(5):479-487.

7. Ende J. Feedback in clinical medical education. JAMA. 1983 Aug 12;250(6):777-781.

8. Ende J, Pomerantz A, Erickson F. Preceptors' strategies for correcting residents in an ambulatory care medicine setting: a qualitative analysis. Acad Med. 1995 Mar;70(3):224-229.

9. Pangaro L. A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad Med. 1999;74(11):1203-1207.

Appendix

Case 1: presented by a second-year medical student during Clinical Fundamentals

CC: Burning with urinationMrs. Jones is a 36-year-old female who presents with the complaint of painful urination for one week. She says that she has had many problems with burning in the past. Sometimes she has smelly urine, increased frequency, and fevers. She may also have occasional vaginal discharge and diarrhea with her burning. She states that she saw some blood in her urine, but is not sure whether it was from her period. The patient is on birth control pills and is sexually active with more than one partner. Her father had diabetes, but she does not think that she has it. You know my sister had some of these problems, and the doctor always gives her an antibiotic.

On her physical exam, she does not have a fever and has a temperature of 36.6 C. Her lungs are clear, and her heart does not have murmurs. She has some lower quadrant tenderness in the abdomen. She does not have any rashes.

Her medications include prozac, Claritin, birth control pills, and calcium tablets. She has an allergy to erythromycin, which gave her vomiting. It is a pretty toxic medication.

She denies any history of intravenous drug abuse or cocaine. She is divorced. She works at a retirement home. She was born in Arkansas, and moved to California for college. She has had multiple sexual partners but does not participate in anal intercourse. She smokes one pack of cigarettes per day and drinks two beers a day.

Her past medical history is significant only for a hysterectomy. She denies any bladder cancer or kidney cancer. She denies any history of sexually transmitted diseases, hypertension or diabetes.

It can be very difficult to evaluate beginning medical students. The instructor should judge whether the presented history of the present illness can lead to a likely diagnosis.

The evaluator should be able to derive the likely structures or organs involved (Is the chest pain from the heart, lung, gut or chest wall?), and the underlying disease process (Is this trauma, infection, metabolic etc.?). Pertinent positives and negatives should be included, and a problem list generated.

Case 2: written up by a third-year medical student in Doctoring 3

CC: DysuriaHPI: Mrs. Jones is a healthy 36-year-old female who presents with complaints of dysuria and frequency. She admits to a history of approximately two urinary tract infections a year that are temporally related to intercourse. She feels that her symptoms are consistent with past urinary tract infections. She denies any malodorous urine, urgency, hematuria, fever, chills or cva tenderness. She has more than one sexual partner, but denies any vaginal discharge, pruritis, or dyspareunia. PMH: She denies any significant past medical history, no diabetes. SH: she denies cigarettes, alcohol, or intravenous drug use. FH: Not contributory. Meds: None. Allergies: None. PE: The patient is a well-developed, well-nourished female in no distress. T=37.5 CP=100 BP=110/60RP=16 Chest - clear to auscultation and percussion Car - Regular rate and rhythm, without murmurs Back - Slight right sided cost-vertebral angle tenderness Abd - slight suprapubic tenderness with positive bowel sounds Labs: A urine dipstick showed 1+ blood, 2+ nitrites, 2+ leockocytes Assessment and Plan: It is likely that Mrs. Jones has an upper urinary tract infection. She has had multiple urinary tract infections in the past and complaints of similar symptoms. Her exam is remarkable for costovertebral angle and suprapubic tenderness and her dipstick is positive. I would suggest a 10-day course of antibiotics.

Semantic competence is a term describing the student's translation of discrete clinical findings into generalized categories that can be used in distinguishing one diagnosis from another.¹ Common semantic axes include acute/chronic, local/systemic, and bilateral/unilateral. The correct use of these more abstract axes in reporting patient data demonstrates that the student is interpreting the findings rather than simply reporting them. For example, in distinguishing the symptoms as indicative of an upper rather than a lower urinary tract infection implies the student has the knowledge and skill to move to the interpreter level.

Case 3: presented by a PGY 2

CC: Sore Throat HPI: Mr. Smith is a 17-year-old with a 4-day history of sore throat and fever. The patient states that he awoke with a sore throat and fever 4 days ago. He admits to an exudate in his throat, fatigue, fever to 101 degrees and tender nodules in his neck. He also admits to severe odynophagia and has not eaten solids over the last 24 hours. He denies any exposure to strep throat or other viral illnesses. He also denies any recent sexual activity, rashes, joint aches or penile discharge. PMH: He denies any chronic illness including diabetes, chronic sinusitis or allergic rhinitis. SH: He denies cigarettes, alcohol or intravenous drug use. His last sexual encounter was 2 months ago. Meds: Tylenol and chloraseptic All: PCN PE: A well-developed 17-year-old male with hoarse voice. T=101 ^oF BP= 105/60 Pulse=108 RP=16 Eyes - Clear sclera. Ears - Tympanic membranes are without erythema and have a normal light reflex. Throat - There is a white exudates with erythema in the peritonsillat region. Tonsils are symmetrically edematous. Neck - Full-range of motion, tender bilateral lymph nodes anteriorly. Tender submandibular lymph nodes. Chest is clear to auscultation and percussion. Cardiac exam reveals a regular rate and rhythm without murmur. Genitals exam shows a penis without discharge. Assessment and Plan: This is a 17-year-old male with pharyngitis. Sources could include strep throat or viral pharyngitis including mononucleosis. Less likely causes would be gonococcal pharyngitis, a peritonsillar abscess or epiglotitis. With his triad of fever, exudative pharyngitis, and cervical tender lymphadenopathy, I would suggest a strep throat culture and a 10-day treatment with erythromycin. Tylenol may control his fever, and I will call the patient in 3 days to be sure that his symptoms are improving.

REFERENCES

1. Bordage G. Elaborated knowledge: a key to successful dianostic thinking. Acad Med. 1994 Nov;69(11):883-885.