This paper introduces a methodology to optimize the trajectory of connected automated vehicles (CAVs) in roundabouts using a two-dimensional point-mass model. We formulate an optimization problem that includes vehicle dynamics and collision-avoidance constraints with explicit representation of vehicle paths. The objective function of the problem minimizes the distance of CAVs to their destinations and their acceleration magnitudes. The methodology also involves a customized solution technique that convexifies the collision-avoidance constraints and employs the alternating direction method of multipliers to decompose the convexified problem into two sub-problems. The first sub-problem only includes vehicle dynamics constraints while the second sub-problem projects the solutions of the first sub-problem onto a collision-free region. The first sub-problem is then transformed into a quadratic problem by redefining its decision variables along vehicle paths. The transformation allows solving this sub-problem with several vehicle-level problems in a distributed architecture. Moreover, we show that iterating between the two sub-problems leads to convergence to the optimal solutions of the convexified problem. The methodology is applied to a case study roundabout with different demand levels. The results show that the trajectory optimization reduced the total travel times and average delays respectively by 9.1% to 36.8% and 95.8% to 98.5% compared to a scenario with human-driven vehicles.