I did the pharm to Med switch, its pretty common. Shoot me a PM if you have other questions, but it was a few years ago back for me.
It is doable but a long battle as you know having to tackle another undergrad. The sacrifice to family, time and energy is something you have to weigh whether it is worth it or not. If it was always your goal as you said, then go for it! I did the same and don't regret it.