Other Publications

Education Columns

“On-the-Fly” Evaluations: Promoting Resident Feedback

Rebecca Burke, MD and Victor Sierpina, MD, University of Texas Medical Branch Family Medicine Residency

Providing feedback is an essential skill that physicians perform regularly. Formal training using validated feedback techniques allows one to become more skilled at giving and receiving feedback. While teaching faculty are typically provided feedback training or literature on the topic, formal resident feedback training is generally sparse, and developing mechanisms to allow for frequent and anonymous resident evaluation of faculty is challenging.

Following an annual Accreditation Council for Graduate Medical Education (ACGME) survey, the University of Texas Medical Branch Family Medicine Residency Program recognized faculty-resident feedback as a quality improvement opportunity. Specifically, residents sought better feedback following assignment completion and requested an enhanced process to ensure confidentiality of evaluations they completed. The program’s chief residents led a resident-only meeting to address the survey concerns, and a faculty-resident retreat ensued to provide formal feedback technique training to faculty and residents, teach cultural sensitivity, and foster closer faculty-resident relationships.

While the residency retreat met its stated goals overall, a paramount issue was raised as to how residents can give routine feedback to their faculty without fear or recrimination. A mechanism to provide regular faculty feedback can help improve teaching skills, create a culture that fosters feedback, and improve sensitivity to unconscious biases. This upward feedback model needs to be anonymous, ensure confidentiality, and promote the spirit of the educational alliance. This allows us to achieve better communication, care for each other as professionals and people, and commit as colleagues to the educational and patient care process. To meet these needs, an electronic “On-the-Fly” feedback evaluation form was created through New Innovations, a web-based software application popular among graduate and undergraduate medical educators.

The secure online housing of “On-the-Fly” evaluation forms allows residents to log in to New Innovations to access and complete an evaluation at any time. After selecting the “On-the-Fly” form, residents choose which faculty member they desire to evaluate. Residents then rate the faculty member on the six ACGME core competencies: patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. Ratings are on a scale from 1 (unsatisfactory) to 9 (superior), and include an option for “insufficient contact.” Residents are able to provide specific comments for each competency evaluated along with overall comments. The evaluation takes less than 5 minutes to complete and is shown below (see sample “On-the-Fly” Evaluation below).

In summary, feedback is always two-sided—there is a giver and a receiver. Developing a mechanism for residents to provide anonymous feedback to faculty on a regular basis guarantees that resident voices are heard and that trainees are not only on the receiving end of the feedback loop. An environment conducive to learning on all levels is achieved when mutual trust and respect are ensured and when feedback is given without fear of retaliation. “On-the-Fly” evaluations promote transmission of high-quality feedback that allows residents and faculty to address learning needs, helps strengthen bonds within training programs, and equips all with skills to use throughout their medical careers.

Further information on feedback techniques can be found at teachingphysician.org.

Options for online software to help evaluate your training program:

New Innovations: https://www.new-innov.com/pub/
One-45: https://www.one45.com/
InstantEval: http://instanteval.com/

 

Ask a Question
AI Chatbot Tips

Tips for Using STFM's AI Assistant

STFM's AI Assistant is designed to help you find information and answers about Family Medicine education. While it's a powerful tool, getting the best results depends on how you phrase your questions. Here's how to make the most of your interactions:

1. Avoid Ambiguous Language

Be Clear and Specific: Use precise terms and avoid vague words like "it" or "that" without clear references.

Example:
Instead of: "Can you help me with that?"
Try: "Can you help me update our Family Medicine clerkship curriculum?"
Why this is important: Ambiguous language can confuse the AI, leading to irrelevant or unclear responses. Clear references help the chatbot understand exactly what you're asking.

2. Use Specific Terms

Identify the Subject Clearly: Clearly state the subject or area you need information about.

Example:
Instead of: "What resources does STFM provide?"
Try: "I'm a new program coordinator for a Family Medicine clerkship. What STFM resources are available to help me design or update clerkship curricula?"
Why this is better: Providing details about your role ("program coordinator") and your goal ("design or update clerkship curricula") gives the chatbot enough context to offer more targeted information.

3. Don't Assume the AI Knows Everything

Provide Necessary Details:The STFM AI Assistant has been trained on STFM's business and resources. The AI can only use the information you provide or that it has been trained on.

Example:
Instead of: "How can I improve my program?"
Try: "As a program coordinator for a Family Medicine clerkship, what resources does STFM provide to help me improve student engagement and learning outcomes?"
Why this is important: Including relevant details helps the AI understand your specific situation, leading to more accurate and useful responses.

4. Reset if You Change Topics

Clear Chat History When Switching Topics:

If you move to a completely new topic and the chatbot doesn't recognize the change, click the Clear Chat History button and restate your question.
Note: Clearing your chat history removes all previous context from the chatbot's memory.
Why this is important: Resetting ensures the AI does not carry over irrelevant information, which could lead to confusion or inaccurate answers.

5. Provide Enough Context

Include Background Information: The more context you provide, the better the chatbot can understand and respond to your question.

Example:
Instead of: "What are the best practices?"
Try: "In the context of Family Medicine education, what are the best practices for integrating clinical simulations into the curriculum?"
Why this is important: Specific goals, constraints, or preferences allow the AI to tailor its responses to your unique needs.

6. Ask One Question at a Time

Break Down Complex Queries: If you have multiple questions, ask them separately.

Example:
Instead of: "What are the requirements for faculty development, how do I register for conferences, and what grants are available?"
Try: Start with "What are the faculty development requirements for Family Medicine educators?" Then follow up with your other questions after receiving the response.
Why this is important: This approach ensures each question gets full attention and a complete answer.

Examples of Good vs. Bad Prompts

Bad Prompt

"What type of membership is best for me?"

Why it's bad: The AI Chat Assistant has no information about your background or needs.

Good Prompt

"I'm the chair of the Department of Family Medicine at a major university, and I plan to retire next year. I'd like to stay involved with Family Medicine education. What type of membership is best for me?"

Why it's good: The AI Chat Assistant knows your role, your future plans, and your interest in staying involved, enabling it to provide more relevant advice.

Double Check Important Information

While the AI Chat Assistant is a helpful tool, it can still produce inaccurate or incomplete responses. Always verify critical information with reliable sources or colleagues before taking action.

Technical Limitations

The Chat Assistant:

  • Cannot access external websites or open links
  • Cannot process or view images
  • Cannot make changes to STFM systems or process transactions
  • Cannot access real-time information (like your STFM Member Profile information)

STFM AI Assistant
Disclaimer: The STFM Assistant can make mistakes. Check important information.