Preclerkship Medical Students’ Use of Third-Party Learning Resources

Key Points Question How and why do medical students use third-party medical education resources? Findings This qualitative study of 58 second-year medical students found that participants used resources in various ways and described multiple benefits compared with in-house curricula, particularly efficiency, clarity, and concision; tensions included navigating resource drawbacks and the perception of an antagonistic relationship between medical schools and third-party resources. Participants suggested that medical schools examine the resources, recommend specific ones, integrate them into the curriculum, and subsidize their cost. Meaning These results suggest that there are opportunities for medical educators to incorporate third-party resources into the curriculum or learn from their strengths as perceived by students.


Introduction
Undergraduate medical education is transforming, with increased reliance on asynchronous, virtual learning 1 as medical students are assuming more control of their preclerkship education. 2One factor enabling this is the widespread use of third-party resources (ie, learning resource not created by the medical school) and the decreased reliance on the medical school curriculum.Although medical students' reliance on third-party resources is well documented, 3 there remains limited understanding of why or how students use them.
The 2022 Association of American Medical Colleges (AAMC) Year 2 survey reported that approximately 70% of students used non-institution-created online videos (eg, YouTube) and other online content (eg, Wikipedia) daily or weekly for medical education information. 3The same study reported that only 41.5% of surveyed students attended in-person preclerkship courses "most of the time" or "often." 3 A single-institution survey of second-year students found that a greater percentage used commercially available online lectures compared with institutional lectures (84% vs 62%). 2 Another single-institution survey found that few students reviewed formal coursework to prepare for United States Medical Licensing Examining (USMLE) Step 1 and instead relied on third-party resources. 4A 2-institution quantitative study comparing student preferences for medical school vs third-party resources found participants preferred third-party lecture length, teacher effectiveness, and material quality. 5ere is a notable research gap concerning students' decision-making processes about when and how to use third-party resources within the context of their school curriculum.If medical educators better understand the students' approach to learning, they may uncover novel opportunities to improve preclerkship education.Therefore, the purpose of this study was to explore why and how medical students use third-party resources.Unlike prior survey-based studies, we used a constructivist approach, drawing on the principles of self-directed learning theory, to investigate the usage patterns and motivations behind medical students' engagement with third-party resources.A constructivist approach aims to understand and construct meaning from participants' perspectives, emphasizing the cocreation of knowledge between researchers and participants through iterative data collection and analysis.

Methods
This qualitative study was deemed exempt by the University of Pennsylvania institutional review board as it posed little to no risk to participants.Participants provided informed written consent.We followed the Standards for Reporting Qualitative Research (SRQR) reporting guideline. 6

Design
We conducted focus groups using grounded theory methodology.Our analysis was informed by our sociocultural worldview and our roles at the time of writing.One of us (E.L.) was a fourth-year medical student.Two of us (J.K. and J.D.) have extensive experience in medical education curriculum development and assessment.Our perspectives informed the study design, implementation, and interpretation of results.

Participants
In the fall of 2022, we recruited groups of second-year, predominantly preclerkship students from a convenience sample of 7 US geographically diverse private and public allopathic medical schools.
These institutions have variable preclerkship curriculum structures (eg, traditional lectures, flipped classrooms, problem-based learning) and approaches to assessment.We partnered with leaders at each school to obtain institutional approval to share study information with or recruit participants by email.Eligible participants had to have used a third-party resource at least once.We accepted

JAMA Network Open | Medical Education
Preclerkship Medical Students' Use of Third-Party Learning Resources students until each institution's focus group reached 10 participants with a goal of 5 to 10 participants per group.Participants received a monetary incentive of $50.

Data Collection
Participants completed an online demographic survey prior to the focus group.Demographic characteristics, including race and ethnicity, were assessed to compare focus group composition with US medical student demographics as assessed by the AAMC Year 2 Questionnaire.We developed a focus group script asking about use of third-party resources in relation to use of school curricula.We asked participants to identify and discuss their reasons for use or nonuse, resource benefits and drawbacks, and what they wished their school knew about the resources (eAppendix in Supplement 1).Focus groups explored participants' use of specific resources on the market, including but not limited to Pathoma, Sketchy Medical, Boards and Beyond, UWorld, AMBOSS, First Aid, and crowdsourced free materials such as the Zanki and Anking spaced-repetition flashcard decks.We conducted a pilot focus group with 6 fourth-year medical students at our institution after which we made small revisions to the focus group script.We revised the script as we conducted focus groups concordant with grounded theory technique. 5We ensured data saturation by using probing questions, exploring alternative views and perspectives, and member checking during the focus groups.Focus groups were held until no new themes were identified.
One study author (E.L) facilitated all focus groups via video conference (Zoom Video Communications) between September 2022 and January 2023.Each single institution focus group lasted 70 to 90 minutes.We recorded and transcribed focus groups using web-based Trint automated transcription software (Trint Ltd).One author (E.L) validated and deidentified transcriptions.

Data Analysis
We iteratively analyzed data using constructivist grounded theory methods, including initial open coding, focused coding, and memo writing (NVivo software version 1.7.1 [QSR International]). 7We continued to analyze the data through discussion, engaging in constant comparison between themes and examination of associations between themes and new data as they were collected. 7We shared results with a sample of participants to ensure accuracy of interpretation.Data were analyzed from October 2022 to February 2023.
Participants described engaging in a cyclical process of hearing about third-party resources, considering multiple factors, and deciding whether to use the resource in question.When participants transitioned to a new block or module, they reentered the decision-making cycle, reexamined the resources they used, and often shifted their practices.We organized our results by the 3 cycle steps: (1) hearing about, (2) selecting, and (3) using resources (Figure).We described the tensions associated with this process and participant-recommended solutions.

Hearing About Resources
Overwhelmingly, participants learned about third-party resources from peers, particularly more advanced students, at their own medical school but sometimes from friends at other schools.Knowledge transfer occurred through informal conversations and school-sponsored events.Knowledge of third-party resources became a form of social capital.Participants aimed to imitate high-performing students' study strategies."I think people identify people that they think do well in the class and try to emulate that….I think [spaced-repetition flashcard app] is 1 resource that has a lot of social pressure around it."(Participant 3, focus group 6) Less commonly, participants described learning about resources from social media or prior to medical school.Rarely did students learn about a resource from a course director or their school.

Selecting Resources
In all focus groups, participants discussed the challenge and stress of navigating third-party resources given time and financial constraints."I felt like I was missing something if I didn't do it.It $250 000-$299 999 5 $300 000-$399 999 4 $400 000-$499 999 0 $500 000 or more 3 (5)   Do not know or prefer not to answer 10 (18) b Percentages may not total 100 due to rounding.In the case of race, percentages total greater than 100 due to individuals identifying as multiple races.
c The other race category contained no specific race categories; it was an option available if participants wished to self-identify as other race not listed.

JAMA Network Open | Medical Education
Preclerkship Medical Students' Use of Third-Party Learning Resources was stressful.It is overwhelming, the number of resources that you have to choose from." (Participant 5, focus group 6) Students relied on peer recommendations to select user-friendly, accurate resources but acknowledged some randomness and trial and error.Interestingly, participants rarely described questioning resource accuracy.Participants weighed many contextual factors when deciding if and how to use a third-party resource.We identified 4 factors that influenced resource selection: the school curriculum, course or module; student's individual priorities; alignment with learning preferences; and cost.We present the factors in the order most commonly described (Table 2).

School Curriculum, Course, and/or Module
A key factor participants considered was whether the in-house curriculum met their needs. Participants

Decision-making cycle
Participants described engaging in a cyclical decision-making process of (1) hearing about third-party resources, (2) selecting resources, and (3) using resources; and then reentering the cycle as the context of their learning changed.
and direct….We've had some lectures that [present]   "We use very different resources depending on the block that we're in.So, we keep accumulating resources as we go.This block that we're currently in, we're only using 3 out of the 5 or 6 resources that we already paid for."(Participant 3, focus group 2) "Some of our systems tend to be taught by 1 lecturer and every lecture is given by that individual.And I find that it's a lot clearer and I can focus on the lecture material…vs a course where each lecture was pretty much taught by a different doctor…a hodgepodge of information.I found it very unclear.For that, I had to watch every [online video resource] to understand that content because it didn't make sense to me Participants believed in-house curricula sufficiently prepared them for end-of-course examinations but questioned its adequacy for Step 1.Others expressed that while the preclerkship curriculum clinical content was interesting, the volume of information and the time required for their other endeavors necessitated focusing on the most important content, often defined as what USMLE tests.
However, participants questioned whether what they were learning would help them become a "good doctor."

Adversarial Relation Between the School and Resources
Participants perceived an adversarial relationship between their school and third-party resources.At some institutions, students were discouraged from using the resources while at others the negative regard was communicated more subtly.Participants believed they needed third-party resources to meet their educational and personal goals, but in doing so, felt guilty they were betraying their school.When participants substituted third-party resources for their school curriculum, they feared missing out on what their school could offer, such as face time with faculty, relationships with classmates, and information specific to their local community."[There is] a document that has been shared by older students and it's based on the lecture schedule and the third-party resources that go with that."(Participant 9, focus group 6) Memorization "For me the biggest benefit of [cartoon-style resource] is it's so easy to memorize large amounts of information.That's why I found it so helpful for micro and drugs.I found that the only way is rote memorization, but then when you have the picture, I could recall it very easily even if it was a lot." (Participant 1, focus group 1) "The first time I used [spaced-repetition flashcard app], I remember doing a question bank on that specific system, and I was really surprised by how well the information was sticking.I was like, wow, everything I know is because I've seen that card.Participants wanted their schools to recognize the merits of third-party resources and create synergy with them, making it more feasible and straightforward to engage with and integrate both learning sources.Furthermore, participants wanted their schools to evaluate and endorse resources to reduce the amount of time and energy students spent trying out and identifying the best resources.Some participants suggested that schools use third-party resources for content delivery and restructure class time to focus on knowledge application and practice (eg, flipped classroom).

Concurrent Cost of Medical School and Resources
Participants commented on resource cost compared with medical school tuition given their belief that most, if not all, students were using third-party resources.Some participants attended school lectures to justify tuition costs.Participants wished their institutions would facilitate the subscription process, pay for or subsidize third-party resource costs.

Discussion
This qualitative study explored medical students' use of third-party learning resources.We found that participants use these resources to fill perceived needs in their medical education and that resources offer multiple benefits compared with in-house curricula.Participants engaged in an iterative and often frustrating decision-making process to select resources.They used resources in user-dependent and context-dependent ways while navigating substantial tensions.Identified themes were similar across schools and largely independent of a schools' preclerkship curriculum structure.
Participants' decision-making processes were inextricably linked to social context and time and financial constraints.Additionally, the school environment and US undergraduate medical education culture factored into decisions.The anxiety to perform well in medical school to be competitive for residency led participants to focus on materials that they perceived helped them achieve this goal.
These patterns align with the concept of bounded rationality and fast-and-frugal heuristics. 8unded rationality is a decision-making model that balances the quality of a decision against the costs of extended deliberation.Fast-and-frugal heuristics are part of one's "adaptive toolbox" and "employ a minimum of time, knowledge, and computation to make adaptive choices in real environments." 9Medical students selected resources based on recommendations from more advanced students, particularly academically successful ones.Students used multiple heuristics to make choices, including the recognition heuristic (preference for the familiar), imitate-the-majority, and imitate-the-successful. 10 While heuristics as mental shortcuts are subject to biases, they ease decisions when time is limited and information incomplete. 8Therefore, the use of these heuristics in medical students' decision-making is reasonable and more feasible than attempting to collect every data point about each available resource.Many participants described engaging in satisficing, finding the good-enough option rather than attempting to arrive at the optimal decision. 11Even so, the selection process was difficult and burdensome, in part because of the large number of options and the high stakes involved, and students desired more guidance from their schools.
Participants' use of third-party resources aligns with self-directed learning principles. 12If participants perceived in-house curriculum shortcomings, they identified third-party resources to meet their needs, frequently selecting resources based on their preferences for content presentation and the specific curricular shortcoming(s).Adult learners perform better and are more satisfied when they are empowered and self-directed in their learning. 12We found students prefer to have options for interacting with content, including video-based lectures, memory devices such as cartoons, spaced-repetition flashcards, and practice questions for self-assessment.The menu of options for content engagement offered by third-party resources enabled students to direct their learning, meet their individual needs for efficiency, gain and retain knowledge, and balance multiple competing interests.Third-party resources may help address the increasing calls to personalize medical student education. 13Our findings suggest there are opportunities for medical schools to enhance their preclerkship curriculum's organization and concision, preferentially incorporating andragogical methods such as content application, memory devices, and spaced repetition.
Positioning third-party resources in opposition to the medical school curriculum generates a false dilemma for students.Participants' resources use ranged from supplementing to replacing in-house curriculum.Many participants wanted to engage with both content sources but felt forced to choose given contextual limitations.The real or perceived adversarial relationship between medical school faculty and third-party resources was emotionally taxing for students, and the guilt some participants experienced using third-party resources surprised us.We recommend that schools explore and evaluate widely used third-party resources.There may be utility in incorporating highquality third-party resources into or alongside the formal medical school curriculum to provide curricular options to students, enhance student learning, and reduce curricular burdens on institutions.If third-party resources are formally integrated into the curricula, schools should ensure

JAMA Network Open | Medical Education
Preclerkship Medical Students' Use of Third-Party Learning Resources

a
Participating schools: Perelman School of Medicine; University of Utah School of Medicine; Albert Einstein College of Medicine; Pritzker School of Medicine; Wake Forest University School of Medicine; University of Colorado School of Medicine; Zucker School of Medicine at Hofstra/Northwell.

"
My go-to is outside resources….I would like to almost entirely use outside resources because I think they have put so much work into them, and I know they're good, and I know they have the material that I need to know." (Participant 7, focus group 4) Scaffolding "I use [cartoon style resource] as a first pass.I'll watch it the night before, the corresponding topics that we're going to go over in class.And then when I go to class, I'm like, 'Oh, like this was mentioned in the class and resource, so it's probably important.'I kind of use that as a way to determine what I think is going to be more important."(Participant 2, focus group 4) "[My] goal is to watch the corresponding [video-based resource] prior to lecture.It helps prime me and provide a framework vs just jumping into lecture."(Participant 2, focus group 5) Search function "If I have a very specific thing that I'm not understanding, I sometimes find it's helpful to type in the name of the topic.(Participant 7, focus group 1) "Sometimes I'll type a subject name in the [resource] search bar and it'll pull up tailored questions to certain topics."(Participant 5, focus group 2) Repetition "The data shows [spaced repetition resource] is the best for spaced repetition and long-term retention of the material.What I'm focusing on is long term retention of everything I'm learning currently and stuff we learned last year.I feel like that's the best way to make sure you're retaining everything you need to know." (Participant 5, focus group 3) Remedies confusion "Any time I've used [video resource], it's when something is not clicking.And the lecture was not helpful to me understanding it.I'm searching the concept to try to get…a succinct way of hearing another person explain it differently."(Participant 7, focus group 1) "For topics that I found difficult and I just wanted like the big picture, I would use [resource]."(Participant 8, focus group 2)

Table 1
expressed dissatisfaction with their schools' clarity, organization, efficiency, and style of content presentation.Many participants desired more concise lectures, increased use of memory devices to facilitate retention, and opportunities to apply knowledge.Students selected third-party resources to meet these needs and address perceived curricular gaps."I find [an online lecture resource] a lot more succinct.Our lecturers are so passionate, and I feel bad…but whenever they start talking about glaciers or water holes and I'm just super tired…which is why I like that little snippet: no intro, no conclusion.I don't have to use extra time where I might zone out because I have a goldfish attention span."(Participant 5, focus group 1) Participants desired customized learning materials, often emphasizing the importance of alignment with USMLE Step examinations."Sometimeslectures aren't getting to the point of what's important about a topic, maybe because it's something [the lecturer is] super passionate about.They're telling you all this interesting, cool stuff about a topic.But in the back of my mind, I'm like, is this going to be on Step?Is this going to be on my exam?I prefer a lecture that's more straightforward Figure.Decision-Making Cycle 20 years of research studies, but I don't know if that's what I have the capacity to remember."(Participant 3, focus group 4) Participants iteratively engage in the decision-making process, altering their use of resources for a new course or when their priorities shifted."Every block, it ends up being a crapshoot at first to figure out which [resource] I like."(Participant 6, focus group 7) Frequently participants settled for what was good enough due to the costs (financial, time, Participants shared that a pass-fail curriculum increased reliance on third-party resources.Even if resources did not align perfectly with school examinations, they were typically aligned well enough to pass.In contrast, pass-fail USMLE Step 1 scoring did not drive participants to rely on their school curriculum."I know plenty of people who never look at the class stuff.They just do [third-party resources]; they pass every time.It's probably the difference between 75 or 80 vs a high 90.The difference between remembering the 1 weird mutation on the slide or not, that you probably didn't need to know anyway."(Participant 10, focus group 3)

Table 2 .
Factors Considered Prior to Resource Selection "Lectures have a lot of extraneous information that is neither clinically relevant nor even covered on the test.I know some people have intellectual curiosity about experiments and things like that.I personally don't.I find it a really annoying experience to listen to a lecture where someone's going off on a ton of tangents or have classmates who are asking a bunch of tangential questions.It feels like a big waste of time when I could watch one of these videos in 10 minutes and get the core facts.And that is a real benefit to how well-organized these videos are." (Participant 1, focus group 1) "[The school gave] a 2-hour lecture list of the names of all the bacteria or viruses you should know.They did not really go into depth very well.And it wasn't an effective way to [learn]…You didn't walk out knowing [the content]; you had to go watch some videos or read some text, do something to actually get it later.It honestly just made it time-efficient to [decide], I'm not going to go to class.I'm just going to watch these videos, and I'll know the stuff I need to know." (Participant 6, focus group 2) The memory hooks really stick with me.I think there's something about a cartoon that tricks me into thinking it's a little bit more fun and engaging, especially when there are very long drug names that otherwise I don't remember.It's nice to have some kind of picture that's weird and sticks with you, and I can remember the drug name so that when I'm in the clinical setting, I don't feel clueless."(Participant 5, focus group 1) "I personally learn well from videos, so I prefer the [online video lecture resource] style of learning.But [reference style resource] has been great because it has everything, and it highlights keywords.So even if you only know a little tidbit of something, you can type it in and generally you'll get a result that puts it into the context that you're looking for.And with the case-based curriculum, I think that it's helpful to have the ability to click into basic science knowledge directly from an article about a disease pathology."(Participant 7, focus group 7) "I did not use [spaced-repetition flashcard app] first year because it did not work for my brain, and it was very overwhelming at times.
." (Participant 6, focus group 3) Individual priorities "[Third-party resources] help me keep my sanity too.In college I burned myself out.In med school, I promised I wouldn't do that.Discovering how to use the third-party resources helped me cut down time to not only do extracurriculars and research, but also have time for myself."(Participant 10, focus group 3) "The thing that's in the back of my mind is AOA, because I think you have to be top 15% of your class, which was motivating me to do better than just pass.In case I decide to go to competitive specialty, if anything can make me stand out is probably AOA in terms of grades.Cost "I didn't want to fork out more money and I just don't have the time or the bandwidth to use all of them to their potential.I decided to stick to the ones that I had rather than like get a bunch and not use them."(Participant 1, focus group 2) "Most people who use [third party resource] have been using it a very long time.If you're willing to commit that much time to it, then you can use it.But if you're not willing to commit that much to it or you didn't start early enough, it just isn't that useful."(Participant 1, focus group 7) "In terms of [reference style resource], I think paying for it was a barrier."(Participant 4, focus group 7) Abbreviation: AOA, Alpha Omega Alpha.

Table 3 .
How Preclerkship Medical Students Use Third-Party Resources "I really liked practice questions to poke holes or [determine] what can I not differentiate?What detail am I missing?Because for me that was way more effective."(Participant 1, focus group 6) Combine multiple resources "I don't watch lecture.If there's a neuro block, I would go in the neuro section of the [spaced-repetition flashcard app crowdsourced] deck, and I would use whatever resource that I preferred, which usually ended up being a combination of [online video lecture resource] and [cartoon-style resource] in order to learn that content.And then afterwards, I'd go through the lecture titles, and if I saw the title of something that I haven't learned or something that didn't look familiar to me, then I would skim through the slides really quick and try to pick out any information that may be relevant on the exam that wasn't covered by the third-party stuff."(Participant 1, focus group 5) Highlight most important content "You could study medicine forever and never truly master it.The good thing about third-party resources is that they pound in high-yield material.Everything it's presenting you with is effectively high yield.When you study based on that philosophy, you then absorb all the high-yield information, which ultimately, is the most important….I found that when I was studying primarily lecture first year, a lot of my studying time was spent reading through lecture and trying to pull out what is high yield and trying to figure out what I really need to know.I think the high-yield stuff should come first."(Participant 4, focus group 5) Map out with syllabus "I scheduled all the [online video lecture resource] for the semester and matched them with what sounded like the right lecture…it helps me to know that what we're going through in lecture actually is going to be relevant on Step and that it'll mean something."(Participant 7, focus group 2) Primary source of learning "We had a lot of people in our class who would sit in the back of the classroom and watch [online video lecture resource] or watch external resources to get the information quickly or in a more efficient format than what they thought class would give them."(Participant 2, focus group 6) "I didn't even go to class for micro because I knew that [cartoon style resource] would help me."(Participant 7, focus group 4) " (Participant 5, focus group 3) Motivation "Seeing that little number [of scheduled spaced-repetition flashcard reviews], it's like, hey, you haven't done this.That is a very big motivator for me, as opposed to how I can always just push off lecture a day or two, but I know that [the flashcards] keep building up if I don't do it daily."(Participant 6, focus group 1)

Table 4 .
Tensions and Solutions Navigating the Use of Third-Party Resources lot of [course directors] believe that their material is going to better prepare us to be better doctors.We all know that Step 1 is not the best test to determine if you're going to be a good doctor.I think part of the curriculum is trying to cater toward the idea that medical school should prepare you to be a doctor and not just take a test.(Participant 5, focus group 5) I completely stopped watching lectures this year.I felt guilty in the beginning because I [thought], this is disrespectful to our professors and there's so much we can gain from their real-life knowledge.But then I told myself, we're going to learn from clinicians on rotations.The purpose of preclinical is to learn basic science.It's still a struggle in my head because I understand the utility of learning from doctors, but right now that's not my focus.(Participant 5, focus group 3) Paying tuition…that's another reason why I want to watch the lectures and be as much a part of the [school] curriculum that I can, because it does seem absurd to be paying to go to medical school and only watching these videos that I could just watch at home online.(Participant 6, focus group 3)