It is best to find a field and topic of study that you love, so if that means going to a different program to do that, then that is what you should do. I know that there are several programs around the country in quantum electronics that are actually within departments of electrical and computer engineering, so it may be possible for you to pursue your interests without changing "majors." The only complicating factor that I see in moving to a physics department is that, depending on the department's requirements, etc., you may need to take more physics-oriented background courses at the start. ECE and Physics curricula have a great deal of overlap, but the two disciplines view some of the research issues from different perspectives. However, if this is the path that you want to take, find a strong program and advisor that will help you navigate this process and you should be in good shape.