This question has four close votes. Back when it had one close vote, I attempted to explain why it was not a duplicate of a similar question. I'd like to ask for some help editing it so it's clearer why it should remain open.
To start with, in my opinion, much of the question is irrelevant backstory. It might be cleaner to strip that out when thinking about the question. That said, as I will discuss below, I think it is a pretty clear duplicate, so it might be better to use the backstory to highlight why it is a new question.
In your question you ask
How hard would it be to apply to switch over to a math department (either in the same university, or a different one), assuming I had taken sufficient math courses, and done very well in them?
The duplicate question asks
For example, suppose someone has enrolled in a computer science phd program. Can he switch over to math(or physics) phd program in the same school later?
I find it hard to see how your question is not a duplicate. You argument that your question is unique seems to be that you would like to see a specific focus on the math/physics interface. I think this is covered in an answer to the duplicate
Other factors that can mitigate or complicate this process: if there's a standard procedure for doing this, if the two departments historically are comfortable with movement back and forth, if the departments are in the same college/school within the university, and so on.
In my opinion, there is nothing special about the math/physics interface. Math and Physics departments sometimes get along and sometimes do not get along. At that level, the answer is really going to depend on the actual departments.