This seems like such a small thing, but it was bugging me until I fiddled around and found a solution.
I have noticed that in my email, the default Reply button was actually Reply to All. We all know the difference and have made the mistake at least once of hitting Reply to All when we just meant Reply. Hopefully for you, it was annoying instead of embarrassing. It seems to effect the Office 365 email app users more than the Outlook users.
Take a look at your email. Up in the right corner, does it say Reply? or Reply to All? If it says Reply to All, you can change that by clicking on the gear up in the top right, selecting Options, then choosing Reply Settings, and select Reply instead of Reply to All. Make sure you click Save, then click the left arrow up at the top left next to Options to get back to your email.