It's all about the angle: Your photos, re-composed

In contrast to other generative image editing solutions, our method consists of two stages: (1) 3D scene and camera estimation and (2) generative inpainting and retouching. By decoupling 3D estimation from image formation, we can faithfully manipulate the scene in 3D and adjust both camera intrinsics and extrinsics. Further, we utilize ML models to understand scene contents and suggest new camera parameters automatically.

In the first step, we use an internal 3D point map estimation model specifically configured to faithfully reconstruct human bodies and faces to limit reconstruction artifacts that would potentially harm identity preservation. For every pixel of the original image, our model estimates a 3D point representing the visible surface patch, and additionally approximates the focal length of the original camera.

Next, we use classical 3D rendering to generate an estimate of the image as if captured with the altered camera parameters. Importantly, we can modify both the camera pose (position and orientation) and focal length, giving us full control over the image formation process.

However, rendering a 3D point map alone is insufficient: when you move a virtual camera "around" an object, you reveal parts of the background that were never captured by the original lens. Essentially, the point map is an incomplete representation of the scene and rendering it from a new perspective always results in "holes." To fill these areas, we use a generative latent diffusion model to complete and correct the rendered estimate. This model was trained specifically for this task using an internal dataset of image pairs with known camera parameters. During training, we estimate the 3D point map of one image and project it into the camera of the second image. The model then learns to reconstruct the second image from the re-rendered first image. At inference time, we employ classifier guidance with regional scaling to faithfully preserve original content, while allowing the model creative freedom to fill in the blanks.

To support fully automatic editing, we utilize ML models to detect the position and 3D orientation of the faces of the main subjects. Together with the 3D point map, this semantic information allows us to compute the camera parameters for the ideal framing. This is particularly useful for portraits. Additionally, images captured with wide-angle front cameras often suffer from strong perspective distortion which can make features closest to the lens appear unnaturally large. To this end, our method automatically detects these distortions and adjusts the virtual camera intrinsics to restore natural, flattering proportions, effectively "stepping back" from the subject after the fact.

This fully automatic solution is now live in Google Photos as part of Auto frame. It seamlessly enhances portraits by using our 3D-aware image editing tool to process eligible photos that contain people. Users can access the re-composed image, which has an automatically adjusted camera viewpoint, as the second rendition option within the Auto frame candidates, making it a single-action improvement to the photo.

This feature is the result of a collaboration between Google DeepMind and Google Platforms & Devices teams. Key contributors include: Thiemo Alldieck, Marcos Seefelder, Hannah Woods, Pedro Velez, Michael Milne, Bert Le, Navin Sarma, Jasmin Repenning, and Selena Shang. Advisors include: Steven Hickson, Claudio Martella, Irfan Essa, and Alex Rav Acha. Special thanks to: Mike Krainin, Jan Stria, Neal Wadhwa, Amit Raj, Mauro Rego, Kita Boice, Dennis Shtatnov, Yuan Qi, Julian Iseringhausen, Peter Zhizhin, Jiaping Zhao, Andre Araujo, Jana Ehmann, Keng-Sheng Lin, Isalo Montacute, Brandon Ruffin, Reginald Ballesteros, and Andy Radin.