Short answer: I think you should stick with it.
Long answer: I'm an aerospace engineer who didn't think physics was her strong suit either. And that's okay for several reasons. First Aerospace Engineering has many aspects to it and not all of them are directly related to physics. I specialized in structures which is based more in material properties and strengths of materials. But there are other areas as well that don't use as much physics or use physics in different ways. Second the physics you take in school vs engineering physics are a little different. Engineering is applied physics which means it is applied to practical real world applications. This means things tend to operate and are analyzed in a manner that is more how you'd expect them to behave. That said, it may not be all physics that you struggle with. There's a lot of different areas, orbital mechanics, circuits, propulsion, design, dynamics and controls, etc. So while one may seem hard to you another may not be as difficult.
Keeping all that in mind, engineering is hard. There will be things that come naturally to you and other things you will have to work at. Physics may be something you have to work harder at. I know I did. But despite that I'm still an aersopace engineer. So keep trying, ask for help, and you'll get there one day.
Best of luck in whatever field of study you pursue, but I hope you stick with your love of space and look into aerospace engineering. I know you can do it!