Anki Anesthesiology

4. Wothe JK, Wanberg LJ, Hohle RD, Sakher AA, Bosacker LE, Khan F, Olson AP, Satin DJ. Academic and wellness outcomes associated with use of Anki spaced repetition software in medical school. J Med Educ Curric Dev. 2023 May 08; 10 :23821205231173289. doi: 10.1177/23821205231173289. https://journals.sagepub.com/doi/10.1177/23821205231173289?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed .10.1177_23821205231173289 [PMC free article] [PubMed] [CrossRef] [Google Scholar]

Anki Tagger: A Generative AI Tool for Aligning Third-Party Resources to Preclinical Curriculum

Zachary Chalmers, Northwestern University Feinberg School of Medicine, 303 E Chicago Ave, Morton 1-670, Chicago, IL, 60611, United States, Phone: 1 3125038194, Email: [email protected] .

Tricia Pendergrast

1 Department of Anesthesiology, University of Michigan Medicine, Ann Arbor, MI, United States

Zachary Chalmers

2 Northwestern University Feinberg School of Medicine, Chicago, IL, United States

Received 2023 Jul 6; Revisions requested 2023 Jul 26; Revised 2023 Aug 1; Accepted 2023 Aug 17.

Copyright ©Tricia Pendergrast, Zachary Chalmers. Originally published in JMIR Medical Education (https://mededu.jmir.org), 20.09.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Education, is properly cited. The complete bibliographic information, a link to the original publication on https://mededu.jmir.org/, as well as this copyright and license information must be included.

Abstract

Using large language models, we developed a method to efficiently query existing flashcard libraries and select those most relevant to an individual’s medical school curricula.

Keywords: ChatGPT, undergraduate medical education, large language models, Anki, flashcards, artificial intelligence, AI

Introduction

ChatGPT is a natural language processing tool that uses deep learning to generate responses to questions from human users [1]. ChatGPT has many possible applications in health care and medical education [2].

Medical students complete much of their preclinical didactic learning outside of the classroom, with the assistance of third-party resources such as Anki flashcard decks, instead of traditional lectures [3]. Anki flashcard decks use the principle of spaced repetition to improve memorization [4,5]. Medical students found Anki flashcards produced for their specific curriculum helpful and believed that these flashcards reduced anxiety. However, most medical students use open-sourced flashcards available online [6]. These decks are maintained by medical students who collaborate using the social media platform Reddit (/r/medicalschoolanki) [7] and through a subscription-based web application that facilitates crowdsourced peer review of flashcard content [8]. Medical students work together to address errors in the flashcards and update them as needed.

Use of crowdsourced flashcard decks eliminates the investment of time required upfront to produce flashcards for each lecture, but these flashcards are not specific to the user’s medical school curriculum [4]. A mechanism to match existing flashcards, created and vetted by medical students within the Reddit and AnkiHub communities, to the learning goals of didactic lectures delivered by medical school faculty members would be less time-intensive for faculty and students. In this research letter, we describe a novel method to efficiently select relevant flashcards from existing Anki decks and associate those cards with individual lectures within the user’s medical school curriculum.

Methods

There are 4 core steps in the workflow ( Figure 1 ). The cards of a target Anki deck are embedded in a large language model (LLM). The gpt-3.5-turbo-16k model summarizes the learning guide into a set of comprehensive learning questions. Cards are presorted for their relevance to the learning question, using the LLM deck embedding, and then gpt-3.5-turbo scores the relevance of these cards to the learning question, which continues until a user-defined query limit for the learning question has been reached. Finally, cards are tagged in the original Anki file, stratified into “highly relevant,” “somewhat relevant,” or “minimally relevant” categories. Technical documentation and scripts are deposited in GitHub [9].

Results

Using the method described above, we selected flashcards from the AnKing flashcard deck that contained 35,152 flashcards and tagged them to our institution’s preclinical curriculum ( Figure 2 ) [8]. We obtained a total of 465 science of medicine lecture guides spanning the 15 system-based modules at Feinberg School of Medicine for the 2022-2023 academic year. For each lecture guide, an average of 13 (range 5-34) summary learning questions were generated by our algorithm. For example, a lecture on central nervous system cancers, might include the following questions: “How do we diagnose and treat gliomas?” and “What genetic syndromes are associated with benign and malignant tumors in the brain?” After generating 4918 unique learning questions, the selection algorithm yielded a total of 21,400 flashcards from the AnKing deck, of which 16,113 were designated as highly relevant to a learning question. On average, 88 (range 11-221) flashcards were selected per lecture. Upon inspection of a sample of lectures, the quality of selections was considered high, with >90% of cards appearing highly relevant. The process developed is highly scalable, with individual lecture guides processed in minutes at minimal computational cost.

Hierarchical tag structure.

Discussion

It is up to medical schools to decide how to adapt to a status quo increasingly defined by student-driven medical education. One possibility is for medical schools to align the student-driven curriculum with the instructor-led curriculum and consider the incorporation of vetted, third-party resources, such as Anki, into didactic learning [3].

Using large language models, we developed a method to efficiently query flashcards in existing widely used libraries and select those most relevant to an individual’s medical school curricula. The feasibility of implementing a ChatGPT flashcard generation into pre-clerkship medical school curricula has not been evaluated and is an area of future study, with algorithmic fine-tuning and prompt optimization likely to further increase the specificity of selections Subsequently, a comparison of medical students’ satisfaction with self-made Anki flashcards compared to ChatGPT-tagged Anki flashcard decks should be conducted.

Abbreviations

Footnotes

Conflicts of Interest: None declared.

References

1. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How does ChatGPT perform on the United States Medical Licensing Examination? the implications of large language models for medical education and knowledge assessment. JMIR Med Educ. 2023 Feb 08; 9 :e45312. doi: 10.2196/45312. https://mededu.jmir.org/2023//e45312/ v9i1e45312 [PMC free article] [PubMed] [CrossRef] [Google Scholar]

2. Ayoub NF, Lee Y, Grimm D, Balakrishnan K. Comparison between ChatGPT and Google Search as sources of postoperative patient instructions. JAMA Otolaryngol Head Neck Surg. 2023 Jun 01; 149 (6):556–558. doi: 10.1001/jamaoto.2023.0704. 2804300 [PMC free article] [PubMed] [CrossRef] [Google Scholar]

3. Wu JH, Gruppuso PA, Adashi EY. The self-directed medical student curriculum. JAMA. 2021 Nov 23; 326 (20):2005–2006. doi: 10.1001/jama.2021.16312. 2785917 [PubMed] [CrossRef] [Google Scholar]

4. Wothe JK, Wanberg LJ, Hohle RD, Sakher AA, Bosacker LE, Khan F, Olson AP, Satin DJ. Academic and wellness outcomes associated with use of Anki spaced repetition software in medical school. J Med Educ Curric Dev. 2023 May 08; 10 :23821205231173289. doi: 10.1177/23821205231173289. https://journals.sagepub.com/doi/10.1177/23821205231173289?url_ver=Z39.88-2003&rfr_id=ori:rid:crossref.org&rfr_dat=cr_pub0pubmed .10.1177_23821205231173289 [PMC free article] [PubMed] [CrossRef] [Google Scholar]

5. Jape D, Zhou J, Bullock S. A spaced-repetition approach to enhance medical student learning and engagement in medical pharmacology. BMC Med Educ. 2022 May 02; 22 (1):337. doi: 10.1186/s12909-022-03324-8. https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-022-03324-8 .10.1186/s12909-022-03324-8 [PMC free article] [PubMed] [CrossRef] [Google Scholar]

6. Rana T, Laoteppitaks C, Zhang G, Troutman G, Chandra S. An investigation of Anki Flashcards as a study tool among first year medical students learning anatomy. The FASEB Journal. 2020 Apr 20; 34 (S1):1–1. doi: 10.1096/fasebj.2020.34.s1.09736. [CrossRef] [Google Scholar]

How I *actually* studied for the ABA Applied Exam [anesthesia oral boards] and practical advice 9 min read

Back in January 2021, I posted about my plans for studying for the ABA Applied Exam [a.k.a. anesthesia oral boards, the-scariest-board-exam-ever, etc]. They were really involved and intense and lofty plans. Like any human, I did not actually follow through on all of them, but hey, all that matters is that I passed [cue applause, firework emojis]. Now, I figured it was a good time to impart all of my wisdom on you as you embark on your journey of tackling this beast of an exam.

If you’re just starting off on trying to figure out how to prepare for this beast of an exam and also curious about the format, please first read my first post on studying for the Applied. It’s actually pretty short and sweet. This entry is meant to be a follow-up on that as well as to provide some practical tips. Keep in mind for those of you reading this in the way future [2022 and beyond] that I took my exam using the Zoom format and thus, do not have advice for the practicalities and details of actually having to fly to North Carolina and meet examiners in real life. Also, the Zoom OSCE excluded regional anesthesia.

Anyway, here’s the part that most of you are curious about – the resources I actually used:

• Ultimate Board Prep [UBP]
• ABA Website [SOE sample questions, OSCE content outline, OSCE scenarios]
• TEE views [the paper that the standard views are from; I’ve created a Google Drive folder of these images that I used to make an Anki flashcard deck to study from… to download, just click on the file and click on the down arrow at the top right part of the screen]
• Rapid Review Anesthesiology Oral Boards
• [Sort of] Yao and Artusio’s Anesthesiology: Problem-Oriented Patient Management

What you’ll notice above is that I didn’t change the resources I used at all. For the sake of completeness, I did include the ABA website resources in this list because literally all I used to study for the OSCE were the content outline and scenarios. And another thing that I want to be totally transparent about is that in my fantasy world where I’m 100% efficient and productive and perfect, I used Yao’s to deepen my understanding and knowledge of anesthesiology… but in the real world, I opened Yao’s like… twice.

How about some practical tips?

Review/study as many scenarios as possible
I was totally overwhelmed with the prospect of studying for the oral boards when I opened the first UBP case and had literally no idea how I would ever remember all the esoteric things in medicine that I had forgotten. Due to the pandemic, my exam was delayed almost a year… so it had been like 2 years since I cared for a pediatric or obstetric patient. I basically felt like I was re-learning all of residency again. Anyway, just remember that these stems are designed to literally contain every possible pathology that you’ve seen maybe zero times. Take a breather. Then take it one case at a time. But actually read, review, and study the cases.

Some people had access to some of the old ABA case stems. I personally didn’t use them because I need explanations to study. I can’t just have the questions and wonder if my answers are right.

Practice with real board examiners
As part of my residency, we were scheduled for mock oral board sessions post-call with the most intimidating attending ever. I still remember my first mock oral board session post call and getting a question about distinguishing between SIADH and CSW and I wanted to curl up into a ball and cry. Needless to say, I didn’t get the question right and to this day, I still have to remind myself distinguishing factors. The point is that you really won’t know how you’re going to do when it comes to the stress of being interrogated by a stone-faced examiner until you actually practice.

Don’t make the mistake of waiting until the last minute to do practice sessions with examiners because it takes time to practice how you answer questions. It doesn’t matter if you don’t have all the material studied/reviewed yet because you’ve literally spent the last decade of your life learning how to study. You’ll get through that part. But none of us have ever had to do an oral examination, so the real challenge of oral boards is learning how to speak the language of anesthesiology.

Learn to answer the questions
As I alluded to in the previous piece of advice, this exam is especially challenging because we haven’t done anything like it. A big part of being successful is learning to answer the question. While I recommend using UBP as a study resource, I do not recommend memorizing and regurgitating the answers verbatim on your exam because you will get cut off before even answering the question. Just answer the question. Be concise. Be succinct. Don’t fill the space with nonsense. The examiners have done this for a while, they can see through your, “Well, I would assess the patient with a physical exam including a thorough hair examination.” [/sarcasm]

“I don’t know.”
Don’t use this statement for every question you encounter. But if you really have no idea, and it’s not a question that you can rationalize through, then move on. Wasting time only means that you might miss out on questions that you know later on. If you’re wondering, yes, I did actually say I don’t know for a least one question.

Use your note-taking time
There is a lot of time to take notes during the long-stem SOE case. Like too much time. So after I outlined the stem [I did end up using the outlining strategy on the real exam that I shared in my last blog post; I found that it was helpful in anticipating questions and organizing the data provided and my thoughts] I did a brain dump. I drew out the different types of tracheo-esophageal fistulas, wrote out the STOP BANG score, wrote out the exam findings that go into a GCS score, and some other random facts that I was sure to forget when asked. Not all of you will need to do things like this… but I have a terrible memory so I needed to write as much down as possible. Just think about how you want to use your note-taking time because there’s a lot of it.

Be honest with yourself [or get an external opinion]
We all have strengths and weaknesses. Some of us are very comfortable with articulating a clinical plan along with our rationale and reasoning. Some of us need more time to collect our thoughts in an organized fashion. Some of us are like deer-in-headlights when asked probing questions about our decision-making. Before embarking on your studying, you need to do a self-assessment and be honest with yourself on whether you will stumble with trying to answer questions in a calm, collected manner when faced with a board examiner, or if you will stumble. If you don’t think you can be honest, do a practice exam with a real board examiner and ask for their opinion. I truly believe that every anesthesia resident completes residency with the medical knowledge to pass this exam, but not every resident possesses the communication skills necessary to pass.

The OSCE
This part was reminiscent of my medical school OSCEs. The interview-type stations can be prepared for with what is provided in the OSCE Example Scenarios document, so study them.

For the monitors and TEE stations, pay special attention to the links to the various links that are in the OSCE Example Scenarios document. For example, on page 7 of the document, there is a link to a sample monitors station video. On page 8, there are links to the 45 second, 2 minute and 2.5 minute TEE image identification and cases. Reviewing all of these links will ensure that you’re prepared for how the monitors will look, how the case will be presented to you, what kind of questions can be asked and the timing of everything.

The stations with TEE and monitors are straightforward if you have reviewed the samples and know what to expect. I will say that the time seems to go really quickly when you’re in the exam.

Zoom exam specifics
The ABA hosts a webinar to give you all the nitty-gritty details of how the exam day will go and some of the other specifics of how the stems will appear on your screen. There’s a technology check email that the ABA will send close to exam day to ensure you have the right specifications for your browser. On exam day, you will have a separate browser window open next to Zoom in order to view the exam materials like the case stems. Additionally, in an effort to mitigate any technological delays, the SOE component is 40 minutes on Zoom instead of 35 minutes. During the webinar, they also discuss that suspicious behavior may lead to termination of the exam [i.e. don’t put all of your resources around your screen and have shifty/squinting eyes trying to read them while answering a question].

The technology component went very smoothly for me.

In the months leading up to your exam date, you will receive two separate times – one for your SOE and one for your OSCE. One of the big benefits of taking the exam over Zoom was that I took a nap between the two sessions. It was the best nap ever. Some of you will probably have some concerns about the technology and how it’ll work for you… I promise that they’ve done these exams for a lot of examinees without too many issues. They’ve figured it out.

If you’re curious as to how my Zoom setup looked, here it is:

Final Thoughts

I thought that my examiners would be super intimidating and I would freeze the moment they asked me anything. But that wasn’t the case. Overall, the exam is fair. The ABA has provided a ton of resources to help you prepare and pass the exam. This all being said, I was 100% convinced I failed after I finished the exam. Luckily it’s done and I’m glad I never have to experience that level of stress ever again.

I hope this is helpful information! If you have more specific questions, feel free to send me a message on any of my social media channels or via email. Good luck!

Share this:

• Click to email a link to a friend (Opens in new window)
• Click to share on Facebook (Opens in new window)
• Click to share on Twitter (Opens in new window)
• Click to share on LinkedIn (Opens in new window)
• Click to share on Pinterest (Opens in new window)
• Click to share on Reddit (Opens in new window)