Ug Transcripts Part 2 - Thoughts From A Program Director

[rmorelan]

I was going to post this on the recent thread but I wanted it to be as open as possible on the forum as it is a rare input from a program director in radiology from my current school of Ottawa no less! Recently the topic of UG transcripts popped up again as a topic on the forum - I think was the one that originally opened discussion on it a few years ago when I noticed a variety of competitive specialties (radiology just being on of them) and looked into it a bit for the various effected specialties. I was curious what the select schools were doing with these, and how it impacted acceptance into the plastics, radiology, emerg, internal....and so on  for the various programs that were using it. Also very interested in WHY exactly this thought UG transcripts should be used at all. In any case,  I wanted people to be aware that if these were being used in any capacity that this was not a surprise to applicants - information is of course the primary reason for the forums existence after all, and the removal of barriers to accessing that information with the hopes of creating a level fair playing field is the goal. With that in mind here is some more information to process obviously from someone that knows:

 

---------------------------------

 

Re: the UG transcripts, can you please post this on my behalf?  Thanks:

 

In Ottawa, we use the UG transcripts as a single data point among many in your application.  It has never made or broken anyone's application.  However, one poster is correct that this is the inevitable result of going to a pass fail system.  Those of you who eventually get involved with Carms file review will discover that Medical school transcripts and dean's letters have precious little, if any useful information on them.  Many of them are essentially a certificate of attendance.  The pass fail system is a tremendous disservice to strong students.  Now, students must 'guess' as to what will help their applications.  Is it research, reference letters, do they have to do electives at every school (impossible!)?  Unfortunately the answer is all of the above and none of the above!   Students are now left to compete in the arena of who can get the 'best research supervisor' or 'elective with the PD'-- this is a measure of networking and luck.  The present system is far from a meritocracy.  I wish that I could say to prospective applicants-- don't worry, just work hard and learn medicine.  If medical schools went back to objective assessment for their students, we would certainly drop the UG transcript requirement.  However, that will not happen in the near future, so we are left with grasping for whatever objective data we can get (it is the ONLY objective data on most applications now!).

 

Ultimately, I encourage you all to apply to our residency program.  We have a very comprehensive, fair and balanced application review process.  Every application is comprehensively reviewed by 3 people and there is no minimum threshold in any one area required.  No system is perfect.  Certainly, in a competitive year, many strong applicants will not get their first choice of program.  If you speak to our trainees, you will see that they come from very diverse backgrounds with varied strengths-- this diversity is integral to the success of our program.  

 

I encourage you all to consider whether or not Canadian medical students would be better served going back to a system of objective assessment.  I think that going back to a merit-based, objective system of assessment would be a step in the right direction.  

 

I direct you all towards a few relevant papers for some additional information that may be of interest (full disclosure-- I co-authored all of them).

 

1.  A study looking at the heterogeneity and lack of objective data among Canadian medical school transcripts:

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4159760/

2.  A study looking at associations between application data and subsequent performance (objective data is not perfect but much better than other data points which have little, if any value):

http://onlinelibrary.wiley.com/doi/10.1111/medu.12234/full

3.  A letter to the editor I wrote on the match in CMAJ:

http://www.cmaj.ca/content/186/13/979/reply#cmaj_el_722374?sid=3007aaba-3b1b-4b4e-ade2-14486c9ace1b

 

Good luck with all of your applications.  Please email me if you have any questions (mmcinnes at toh dot on dot ca).

 

Matthew McInnes MD FRCPC

Diagnostic Radiology Residency Program Director University of Ottawa

Diagnostic Radiologist at the Ottawa Hospital

Associate Professor of Radiology University of Ottawa

 

-----------------------------

 

allow me to also say that the program here at Ottawa is excellent, and certainly one of the best programs in Canada for training. All biases aside I do find it to an excellent training opportunity

[Let'sGo1990]

Thanks for posting this.

 

Great to hear directly from a PD. Pretty awesome he took the time to do that. 

[ralk]

Thank you for posting this, it's great to hear directly from program directors.

 

I agree that the current system is probably too subjective and that the mechanisms by which medical students can improve their chances for competitive residency programs are quite unclear. However, while it's gratifying to know that program directors recognize the problems with the current system, Dr McInnes' statement provides virtually no rationale for the use of undergraduate transcripts.

 

In a system where medical school GPA is a major factor in getting a good residency program, it's not surprising that it has some positive correlation with residency performance. If you tell motivated people to do something, they'll do it. But that doesn't mean doing well on medical school tests makes you a better physician. Doing well on tests often involves a lot of detail-oriented cramming which is typically forgotten soon after the test - hardly useful for residency. Moreover, it's a common refrain that 50% of the thing you learn in medical school will be obsolete by the time you start practicing. More than a few physicians have told me how useless pre-clerkship marks are to the long-term knowledge development necessary to be a capable doctor. Dr McInnes stresses the benefits of a graded system in medical school, but neglects to mention the negatives associated with such a system. Students killed themselves trying to get amazing marks, stressing about every little test, but it didn't necessarily make them significantly better physicians. And it contributed to a high-stress environment that pushed more than a few students too far.

 

That's why I think the current pass/fail system is a marked improvement. I don't have to worry about getting a 90+ on every exam - I can be satisfied with an 80 or even a 70, allowing me to focus on the overarching concepts or critical details to practicing rather than trying to learn every little minor detail I'll forget anyway unless I happen to specialize in that exact field. More importantly, I can spend my time and energy on more productive pursuits. I can do meaningful research. I can spend more time in clinical settings. I can work with student groups to contribute to learning - both my own and my fellow students' - at my school. I can volunteer in my community. Not only will these things likely make me a better physician, they'll help my career development into and beyond residency. When it comes to fulfilling the CanMEDs competencies, doing well in school gets me "Medical Expert" only - the things I get to spend my time on because we have the pass/fail system let me work on all of them.

 

Simply put, the graded system made it easier for residency directors to select good physicians - the pass/fail system makes it easier for medical students to become good physicians.

 

Note that the research provided by Dr McInnes makes no mention of the value of undergraduate marks, only medical school marks. No evidence has been given which indicate the usefulness of undergraduate marks. The only argument put forth is that undergraduate marks are objective, but this is taken for granted that undergraduate marks are, indeed, objective. They are quantitative, certainly, but quantitative is not the same as objective. There is incredible variation between universities, programs, and courses taken in undergrad. Students grow and change considerably from the time they start undergrad to the time they're applying to residencies. For older applicants, this is especially true. What a potential resident did at 18 is not likely reflective of their current capabilities or attitude, positively or negatively. I've said it before and I'll say it again - undergraduate marks are not objective measures of potential competence as a resident, they just give the illusion of objectivity because they happen to be quantitative.

 

Lastly, the concerns Dr McInnes presents could be addressed in other manners that do not have the downsides of a graded medical undergraduate system. Residency programs could post exactly what they consider when selecting residents and how much each element is worth. One program director here at Western gave a presentation where he laid out exactly the system they use, what they consider, how they consider it, and how much each element contributes to their residency selection process. Every program could post this on the CaRMS website. Virtually none do, and Ottawa's radiology residency program is no exception. If this is really a concern Dr McInnes has, I would encourage him to provide a fully transparent rundown of his program's selection process on CaRMS. As for more objective measures, the linked study puts USMLE scores as basically equivalent to Med School GPA in terms of predictive value. There's no reason we couldn't have a similar test in Canada. Heck, we could just insist on the USMLE here as well (is there any reason individual programs couldn't ask for that right now?). That would create some of the same problems having a graded system did, namely a focus on relatively unproductive studying, but preparing for one test is much less onerous than preparing for multiple tests over several years.

 

Dr McInnes presents using undergrad transcripts as an attempt at addressing flaws in the current system. I would argue that using undergrad transcripts not only does not address those flaws, but introduces additional issues to the residency selection process. More importantly, because using undergraduate transcripts are largely a reaction to the move away from the graded system in Canadian medical schools to a pass/fail system, their use has become less a solution than a protest against previous changes. Rather than mourning for a past system that was abandoned for good reason, we should be finding new ways to improve the current system - and program directors like Dr McInnes hold the most power in moving us in that direction.

[B3.572A]

I second ralk's idea of mandatory USMLE scores for CaRMS. Is there any talk about this possibility? I realize that there may be some politics involved because CaRMS is US-based, but we have already used MCAT for medical school admissions.

[rmorelan]

I second ralk's idea of mandatory USMLE scores for CaRMS. Is there any talk about this possibility? I realize that there may be some politics involved because CaRMS is US-based, but we have already used MCAT for medical school admissions.

well that is the first major problem - it is a US test, and the content of that test is not in line with Canadian medical school training. The two systems are not perfectly equivalent. Plus we would have yet another factor added that isn't directly measuring you ability to do clinical medicine and what happens in you decide in your 3rd year to do a competitive field (a lot - my school said the majority only really decided their fields in 3rd year) but haven't written that test.

[ralk]

well that is the first major problem - it is a US test, and the content of that test is not in line with Canadian medical school training. The two systems are not perfectly equivalent. Plus we would have yet another factor added that isn't directly measuring you ability to do clinical medicine and what happens in you decide in your 3rd year to do a competitive field (a lot - my school said the majority only really decided their fields in 3rd year) but haven't written that test.

 

Perfectly valid reasons for programs/specialties not to unilaterally adopt the USMLE. Also great reasons for programs/specialties not to unilaterally adopt looking undergrad transcripts. Undergrad work does not perfectly align with Canadian medical school training, particularly for those of us who took non-standard premedical degrees, it doesn't measure your ability to do clinical medicine in the slightest, and occurs well before applicants decide they want to do a competitive field - heck, much of it may happen before an applicant had decided they wanted to do medicine.

 

I'm not saying adopting the USMLE would be an ideal solution, far from it. Obviously a made-in-Canada test that all students took for all specialties would have more value, though that'd be difficult to implement. Yet all the reasons for not adopting the USMLE apply doubly to undergraduate transcript usage, and at least the USMLEs have some evidence behind their worth as a metric. I get not wanting to use the USMLEs for many reasons, including the ones you list - but that does not, in any way, justify the use of undergrad transcripts.

[GrouchoMarx]

It is refreshing to read about a program director's challenges regarding our currently vague residency selection process.

[rmorelan]

It is refreshing to read about a program director's challenges regarding our currently vague residency selection process.

 

I would agree Actually that is the one thing I am really glad can get across - this is simply not at all an easy thing for programs to do. Selecting people with the information they are provided now is rather challenging to say the least and I would suspect the collective system is fragile to a variety of biases.

 

One other point I guess - like it or not the programs don't have objective criteria for selection. This is a major problem in their eyes and I don't think that is surprising. One way or the other that problem will have to be addressed. Time and time again when an issue is ignored or buried it comes back to bit us in the rear because we failed to take leadership role in the first place. We didn't fix the doctor shortage and we were left with alternative fields rising up in the vacuum left and a surge in medical school applicants belatedly. Now we are dealing with the over reaction and a tighter and tightening job market. We don't proper police ourselves and there starts to be very public redresses towards our shortfalls and questions about whether we should have that right. We didn't address doctor salary issues, and held back on pushing for cost cutting measures in the hospitals even when we knew where those cuts could be done relatively painlessly - so salaries were frozen, control of that process was lost and we look less than professional. So on and so on. Ignore this problem and don't be surprised that UG transcripts (which no one thinks is the optimal solution here) use continues to expand - not as critical response to the pass/fail system but rather an attempt at filling a rational need (in the end more people will apply to things than can be accepted. Not everyone will get what they want or go where they want to go. The question is how would you like that process to work? What criteria should be valued most highly in the process? How do we make sure there are safeguards so your hard work isn't wasted because your competition was lucky etc?)

 

Bottom line - never be passive in any process. Learn, think and act

[shady]

That was a good read. Thank you for posting it!

[grapz]

Hi rmorelan,

 

In your previous post in the other thread, you had mentioned that some program directors you talked to ( I assume ottawa since you are a resident there ) hold UG transcripts in high regard and use it as the main selection criteria to determine if you get a interview or not. Having read this post from your PD, what are your thoughts on this?

[rmorelan]

Hi rmorelan,

 

In your previous post in the other thread, you had mentioned that some program directors you talked to ( I assume ottawa since you are a resident there ) hold UG transcripts in high regard and use it as the main selection criteria to determine if you get a interview or not. Having read this post from your PD, what are your thoughts on this?

 

well it wasn't just Ottawa, and it wasn't just radiology. This is actually an area I am kind of interested in (I am currently doing a Master's in Education at Ottawa - with a subfocus on Health Professional Education). How we decide who gets into medical school, and how gets into what residency program is something I think we should all be aware of and concerned about

 

I think personally that more information is better in the selection process. Like many residents around the country I am involved in selecting the next group of residents coming after me. That means we familiar with the challenges of figuring out how to rank people in our current no objective criteria world.

 

Have to say it really sucks. Everyone looks the same. Everyone passes everything, were members of the exact same clubs, did similar research amount for the most part, and letter of references are hard to interpret (and are all invariably glowing). It is all grey and grey - no contrast to see anything clearly. Since most schools only interview 1/3 to 1/2 of the people that apply there seems to a lot of exclusion for reasons that are unclear to applicants. So unless you are one of the few super applicants or obviously subpar (at best the top and bottom 10%) then there is a lot of luck involved.

 

So people spend years preparing, trying to figure out what we want for selection, make it their life's mission as it were and STILL miss the mark for reasons that are vague and unclear. The most stressful part of all this I found as medical student was simply not REALLY understanding what anyone wanted, so I ended up just hoping that by following my personal interests that it would all work out. It did - but it didn't for some very close friends of mine. People waste so much time really doing things they hope will work but don't. I am reminded that one of the best ways to stress out a type A personality is to give them a clear worthy goal and give them entirely vague criteria for getting there (watch all the premeds on here endlessly calculating odds etc - now make all of that simply impossible, hide all the rules and make everything rumour, and see what happens. That is kind of like CARMS).

 

Would medical school grades help? More objective data is better than less, and something current and relevant seems appropriate. Even the old honours/pass/fail system helped without creaming everyone. I think we need some form of balance between the highly competitive world we had before and the wishy washy world we have now which causes it own types of stress/wasted efforts (like forcing non research types but great clinical people to pump out research over and over again at the expense of studying the material) and creates an unfair playing field of vague subjective evaluation (by the way - you really, really need to make sure your carms stuff is polished well. Presentation in our current environment is quite critical - you need to very quickly appear to have the right stuff).

 

There are problems though - because we are in a pass fail system no one is as focused on testing anywhere (who cares if a question on a test is vague etc - everyone passes anyway). Are all school's grades equivalent? What about Mac - they don't even have grades in the traditional sense.

 

To be honest I really would favour some form of objective across Canada testing system prior to CARMS. I think it would better focus people and also schools, reward directly students that work hard at studying (as opposed to now where all you get is the same pass as the worst student in the class). At least that would be fairer. I don't see that happening anytime soon, so I wouldn't be surprised that we get further requests for UG transcripts or any other form of objective evaluation they can find. 

[ralk]

well it wasn't just Ottawa, and it wasn't just radiology. This is actually an area I am kind of interested in (I am currently doing a Master's in Education at Ottawa - with a subfocus on Health Professional Education). How we decide who gets into medical school, and how gets into what residency program is something I think we should all be aware of and concerned about

 

I think personally that more information is better in the selection process. Like many residents around the country I am involved in selecting the next group of residents coming after me. That means we familiar with the challenges of figuring out how to rank people in our current no objective criteria world.

 

Have to say it really sucks. Everyone looks the same. Everyone passes everything, were members of the exact same clubs, did similar research amount for the most part, and letter of references are hard to interpret (and are all invariably glowing). It is all grey and grey - no contrast to see anything clearly. Since most schools only interview 1/3 to 1/2 of the people that apply there seems to a lot of exclusion for reasons that are unclear to applicants. So unless you are one of the few super applicants or obviously subpar (at best the top and bottom 10%) then there is a lot of luck involved.

 

So people spend years preparing, trying to figure out what we want for selection, make it their life's mission as it were and STILL miss the mark for reasons that are vague and unclear. The most stressful part of all this I found as medical student was simply not REALLY understanding what anyone wanted, so I ended up just hoping that by following my personal interests that it would all work out. It did - but it didn't for some very close friends of mine. People waste so much time really doing things they hope will work but don't. I am reminded that one of the best ways to stress out a type A personality is to give them a clear worthy goal and give them entirely vague criteria for getting there (watch all the premeds on here endlessly calculating odds etc - now make all of that simply impossible, hide all the rules and make everything rumour, and see what happens. That is kind of like CARMS).

 

Would medical school grades help? More objective data is better than less, and something current and relevant seems appropriate. Even the old honours/pass/fail system helped without creaming everyone. I think we need some form of balance between the highly competitive world we had before and the wishy washy world we have now which causes it own types of stress/wasted efforts (like forcing non research types but great clinical people to pump out research over and over again at the expense of studying the material) and creates an unfair playing field of vague subjective evaluation (by the way - you really, really need to make sure your carms stuff is polished well. Presentation in our current environment is quite critical - you need to very quickly appear to have the right stuff).

 

There are problems though - because we are in a pass fail system no one is as focused on testing anywhere (who cares if a question on a test is vague etc - everyone passes anyway). Are all school's grades equivalent? What about Mac - they don't even have grades in the traditional sense.

 

To be honest I really would favour some form of objective across Canada testing system prior to CARMS. I think it would better focus people and also schools, reward directly students that work hard at studying (as opposed to now where all you get is the same pass as the worst student in the class). At least that would be fairer. I don't see that happening anytime soon, so I wouldn't be surprised that we get further requests for UG transcripts or any other form of objective evaluation they can find. 

 

I love hearing the frustrations both you and Dr McInnes have about the current system, and by sharing those opinions publicly you've done much more than most involved in process have done in the past. But one-way communications, fraught with platitudes and lacking any meaningful action plan aren't going to result in change. Just like you, I care about how our residency system (and more broadly, our medical education system) is set up and I want us as a profession to have thorough discussions about the problems with that system. But saying what we don't like isn't enough - we need solutions and feasible plans to implement them.

 

I gave several options to address the complaints listed by you and Dr McInnes, some of which could be done unilaterally by individual programs, including your's. I would love to hear more about what your program is doing to address the current issues, or what you'd like to do but can't.