It may be cause of the automatic rebuild and restart is Auto build and refresh browser after saving changes option under the Tools -> Options -> Projects and Solutions -> ASP .NET Core settings.

Just set it to None or any other option than Auto build and refresh browser after saving changes.

Reference: https://stackoverflow.com/questions/61924591/visual-studio-2019-disable-auto-building-solution-on-save