Nuclear fission and fusion reactions are the examples of conversion of mass into energy. Can we say that strictly speaking, mass is converted into energy even in the exothermic chemical reaction?

Yes, mass is converted into energy in an exothermic chemical reaction also.
But the mass change in a chemical reaction is about a million times less than in a nuclear reaction.