The stone flies of tangentially to the circle along a straight line at the point where the string breaks. It is due to the inertia of direction. When the string breaks, the force acting on the stone ceases. In the absence of force, the stone flies away in the direction of instantaneous velocity which is along the tangent to the circular path.
What happens to a stone tied to the end of a string and whirled in a circle if the string suddenly breaks ? Explain why