Controversy over new FM Structured LOR Rubric

[ana_safavi]

So, it appears that as of this year(?), family medicine programs will be evaluating applicants based on structured LOR rubrics instead of traditional LORs. A few days ago, a sample rubric was posted on twitter.

It was widely criticized for being discriminatory and/or otherwise problematic.

In response to the backlash, CaRMS and the CCFP rushed to distance themselves from it. We still do not know exactly who was involved in drafting the rubric, but it appears to be the severely misguided brainchild of the AFMC and all 17(?) family medicine program directors, who have pledged to revise it in view of the feedback they've received from Canadian MedTwitter.

If they still plan to roll these out for the 2021 match, the revisions presumably need to be completed by November 23 (the date that program descriptions for the R-1 match go live). This would not seem to leave enough time to fix the multiple issues identified with the sample rubric.

Thoughts? Is the FM SLOR salvageable? Or should they just scrap it and start over from scratch for 2022?

 

[1D7]

I find it amusing that there's a drive in the first place to standardize LORs so that they look similar to med student evals.

Honestly if they removed a few words here and there like "tearfulness" I doubt there would be much to be outraged about.

[ellorie]

It strikes me that the approach here in general is not ideal - it seems to be a way of asking the question "does this applicant have flaws/vulnerabilities?"  Except that that is inherently the wrong question, because every medical student has flaws and vulnerabilities.  Maybe some people succeed in never showing them at work - but I'm sure lots of good doctors don't always succeed at that.  And asking these kinds of questions just encourages people to hide things rather than asking for help.

In my opinion, the information they should be trying to elicit should be whether the applicant is able to acknowledge, reflect on, and work on their flaws, recover from mistakes, and manage their moments of vulnerability in a professional way.

I don't see why they can't ask questions like "How does this applicant handle emotionally charged clinical situations?" or "How does this applicant handle conflict with team members?" or "How does this applicant handle feedback?"

My first clinical work-related cry was on my first day of clerkship and my last one was...three weeks ago I think.  I'm better at managing moments of weakness now, but honestly this is no thanks to a significant cohort of supervisors who clearly thought that displaying any emotion was a sign of weakness and unprofessionalism.  I hope we can change this toxic medical culture, because it does nobody any good.

[chateau22]

It's embarrassing that they claimed to have involved many stakeholders in this including all 17 program directors.

 

How did they get it so wrong then?

 

I respect them trying to fix this, but they also havent directly apologized either for the massive flaws in this form.

 

Perhaps they should've spent more than a few months before springing something completely unknown to applicants. 

[chateau22]

Also where is the text in this form?

Are they really reducing family medicine to text boxes and checklists when every other discipline is still using reference letters which can show more about an applicant? 

A major change like this should've had more time invested (and those massive flaws may have been avoided) and given applicants more of a heads up because there were definitely people who asked for letters and now will have to go back to their referees and ask them to fill out this form instead 

[chateau22]

They definitely need to scrap it for this year - it's clear they need more time fixing this, and hopefully they realize liaison with 17 program directors is not enough and get a wider net with more family physicians (who clearly picked up the flaws right away)

But they probably won't do that. They'll just shove it down our throats, throw their hands up in the air and say "Too bad". Clearly they didnt want to invest more than a few weeks into this. Embarassing for family medicine - do better.

[IMislove]

3 hours ago, chateau22 said:

They definitely need to scrap it for this year - it's clear they need more time fixing this, and hopefully they realize liaison with 17 program directors is not enough and get a wider net with more family physicians (who clearly picked up the flaws right away)

But they probably won't do that. They'll just shove it down our throats, throw their hands up in the air and say "Too bad". Clearly they didnt want to invest more than a few weeks into this. Embarassing for family medicine - do better.

They have, and now it has moved to the 2022s CaRMS year. They say it is evidence based to used standardized LORs, as to what the evidence is I haven’t searched for, Sure someone could do a lit review for it hah. So I suppose it is just a delayed issue now, guess we will see how next year pans out.

[Intrepid86]

I like how they listed "spirit of adventure" as being necessary to do rural medicine. The people who designed this rubric obviously practice in an urban area. 

[chateau22]

4 hours ago, IMislove said:

They have, and now it has moved to the 2022s CaRMS year. They say it is evidence based to used standardized LORs, as to what the evidence is I haven’t searched for, Sure someone could do a lit review for it hah. So I suppose it is just a delayed issue now, guess we will see how next year pans out.

Really? According to Twitter, they're trying to make an updated version and as of now, it's still being used for 2021 graduating class

EDIT: Saw the update of being postponed to the 2022 year - great advocacy by FMs, hopefully they take advantage to tweak the form  

[shikimate]

A standardized form would have been good choice if you are hiring a large number of relatively uniform, low to medium skill workers (eg. warehouse employees, cleaning crew, laborer) as it reduces burden on HR and can screen for people with relatively uniform basic level of traits and skills. Also these types of jobs are mobile and a worker is typically not expected to stay with the company, or even the job/field, forever.

To hire a FMD who will likely be practicing for the next 30 years and who will likely have significantly higher individual impact than the aforementioned jobs, a narrative LOR would provide more nuanced response. If someone is not willing to make the effort to write a narrative LOR (rather than take 2 min to make checkmarks on this list), then probably he/she doesn't trust the candidate as much as you think. 

So a checklist would have been good as a screening tool to immediately flag candidates, say for lack of clinical or procedural skills. But some other aspects such as how to handle an emotionally charged encounter also requires a narrative approach.

[Findanus]

Seems backwards. 

There is push to standardize things that shouldn't be standardized, like interviews, and to ignore things that should be, like grades for instance.

[chateau22]

1 hour ago, shikimate said:

A standardized form would have been good choice if you are hiring a large number of relatively uniform, low to medium skill workers (eg. warehouse employees, cleaning crew, laborer) as it reduces burden on HR and can screen for people with relatively uniform basic level of traits and skills. Also these types of jobs are mobile and a worker is typically not expected to stay with the company, or even the job/field, forever.

To hire a FMD who will likely be practicing for the next 30 years and who will likely have significantly higher individual impact than the aforementioned jobs, a narrative LOR would provide more nuanced response. If someone is not willing to make the effort to write a narrative LOR (rather than take 2 min to make checkmarks on this list), then probably he/she doesn't trust the candidate as much as you think. 

So a checklist would have been good as a screening tool to immediately flag candidates, say for lack of clinical or procedural skills. But some other aspects such as how to handle an emotionally charged encounter also requires a narrative approach.

Agreed. It just will be easier to back up with FM when they realize they just need a checklist filled vs. A reference letter 

[ana_safavi]

The advantage to standardized LORs is that we will finally have data available to quantify the effect of discrimination (on the basis of sex, race, family status, and disability) on applicants in the residency selection process.

This is precisely what organizations like #MedicineToo need in order to prove (and thus combat) discrimination, harassment, and retaliation in medical education.

What I wasn’t expecting is that the rubric itself would be written in such a way as to be discriminatory on its face.

At least they’ve revealed what we always knew.

[guy30]

I think this highlights the challenge of not having things like grades or exam scores to compare candidates with each other on during CaRMS. 

They're trying to introduce some sort of objectivity to the CaRMS process when the entire process is subjective and therefore open to things like discrimination, bias etc.

[ArchEnemy]

8 minutes ago, guy30 said:

I think this highlights the challenge of not having things like grades or exam scores to compare candidates with each other on during CaRMS. 

They're trying to introduce some sort of objectivity to the CaRMS process when the entire process is subjective and therefore open to things like discrimination, bias etc.

To say that the process is entirely subjective is also untrue.

There are many ways other than grades to show abilities objectively, such as research productivity, awards, or even leadership positions while in med school.

[1D7]

4 hours ago, ArchEnemy said:

To say that the process is entirely subjective is also untrue.

There are many ways other than grades to show abilities objectively, such as research productivity, awards, or even leadership positions while in med school.

IMO those are mostly subjective or reflect gamesmanship, even awards to an extent (some are influenced by a nomination process).

[Snowmen]

In the mean time, Quebec doesn't even require LORs for FM and holds a single common interview for all french FM programs in the province. Amongst other news, FM programs in Quebec have not imploded. Weird.

[robclem21]

On 11/10/2020 at 1:46 PM, ana_safavi said:

The advantage to standardized LORs is that we will finally have data available to quantify the effect of discrimination (on the basis of sex, race, family status, and disability) on applicants in the residency selection process.

This is precisely what organizations like #MedicineToo need in order to prove (and thus combat) discrimination, harassment, and retaliation in medical education.

What I wasn’t expecting is that the rubric itself would be written in such a way as to be discriminatory on its face.

At least they’ve revealed what we always knew.

While I don't think this standardized LOR is a good idea by any means, and some of the ideas presented in this form as you have posted it are a bit ridiculous (as others have mentioned),  I fail to see how it is discriminatory... Maybe it is just me, but nowhere on this form do the ideas of sex, race, family status or disability even remotely appear. There is also nothing here about dealing with harassment in medical education.

[ellorie]

On the face, it isn't - but we live in a society where the same behaviour can be perceived very differently based on who it is coming from.  For example, behaviour that would be "confident" or "assertive" from men is more likely to be read as "bitchy" or "aggressive" when it comes from women.  Especially women of colour.

So these biases could certainly come out in a free form recommendation letter, but when these kinds of things are explicitly asked about, it does make it more likely that gender or race will contribute to the perception that someone has had these "lapses".