It is never to late to become an engineer! I encourage you to pursue your interest and explore what options are available to you. Your strong skills in math will certainly support your pursuit of an engineering degree. I wouldn't let chemistry hold you back.
First, consider contacting the engineering department at your current school- to review how many of your undergraduate classes would count towards a BS in Civil Engineering. Most likely, you have taken a good portion of the first/second year requirements already and another 4 years would not be necessary. Secondly, graduate programs in engineering often have students from other disciplines. However, given your background in Geography, there may be some undergraduate classes that would be required in order to transition into a Civil Engineering graduate program.
It is hard to recommend which path to take, without knowing more about your specific situation. I can offer a few pieces of advice during your decision making: 1) Make a Pro/Con list, actually write them down- it is amazing how easy some decisions are when you see it on paper! 2) Choose an engineering program with a good intern/co-op program to gain some real-world experience while you finish your degree. 3) If financial support is a major decision-maker, investigate potential scholarships through organizations (e.g. Society of Women Engineers).
Good luck to you!
Kristin Malosh, P.E.