If you enlarge a rectangular selection you'll get rounded corners, click on the Current Selection icon on the Selection Tool ribbon to get sharp corners.

If you want to round out square corners go to Selections>Modify>Smooth.