As people have mentioned in other EM (5 years) vs. FM+ EM threads, it's not entirely practical to split your work between both family medicine and emerg at the same time. Instead, it sounds like most FM+EMs will work in emergency medicine for a given time, while having the option to practice FM down the road (if you become tired of the shift-work, if you're feeling burnt-out etc.)
It also sounds like you have the desire to work/live one day in Toronto. As a FM+EM working in emergency, there may be less opportunity for future work in Toronto, especially in downtown academic centres. Would you be okay with working at a community site, or even 1-2 hours outside of Toronto? Would these community locations hinder your ability to conduct the research that you're interested in? And as others have mentioned, FM+ EM has become very competitive. If you were not able to get into this stream, would you be satisfied working entirely as a family physician?