Engineering is a versatile field and if you like physics and math instead of memorizing chemical formulas, there are still jobs in chemical engineering that may interest you. Chemical processing and chemical manufacturing related jobs typically involve more organic or inorganic chemistry but can also include reactor design and processing which involves math and physics. If you’re designing instruments or involved in other fabrication processes there may be less chemistry involved.
Since chemistry is often considered the ‘central science’ it’s probably good to have a basic understanding of it, regardless of what type of engineer you’d like to be. If you really dislike chemistry and don’t want to pursue chemical engineering, then other options like mechanical engineering or electrical engineering may be of interest. My advice is to take as much math, physics and chemistry to get a solid foundation (along with your engineering classes). Physical chemistry might be more of interest to you than organic chemistry if you don’t like memorizing formulas.
As for health hazards, it’s all about ensuring the proper handling of equipment and chemicals regardless of the job you have. Most companies are very responsible about providing proper controls and conditions for chemical related work. As for your question about if you have to always work in the lab- the answer is no. There are many examples where engineers do theoretical calculations and cost analysis in addition to designing processes or systems rather than actually carrying out the experiments themselves. Often you work as a team, especially in large companies. You might also want to search online for “what do chemical engineers do?” to find a lot more detail that may give you some ideas.