I just did this this year! It's actually not too tricky. I went from Scotiabank/BNS to RBC.
First had to meet with the RBC advisor and apply for their package. They ask you which credit cards you will close if you sign up for their package. I said none, and that seemed fine. Got approved, sent proof of enrolment, then had to do some other stuff next to make sure I don't get dinged by fees.
Upgraded BNS account to their Ultimate package, then downgraded the Gold Amex to the normal Amex, but kept the Passport VI. The Ultimate package gets you the annual fee waiver for the Passport VI