If you add a Rectangle Widget to your Composition and then head over to the rectangle tab within the Property Panel, you can update the Bevel Style and Bevel Size in the "Shape" property box. If you set Bevel Style to "Outside" and then adjust the Bevel Size, you will get rounded edges.
Update the Rectangle Widget's Bevel Style and Bevel Size in the Property Panel
The other way you can do this is to adjust the Group (folder) that houses your rectangle. Select the Group (folder) that houses your rectangle in the Composition Tree, move over to the Effect tab in the Property Panel, and activate the "Clip Children" option in the "Render Options" property box. Then you can adjust the Border Radius. See the image below.
Activate "Clip Children" option in the "Render Options" Effect box