Written by Kevin Hopkins, Computer Vision (AI/ML) Intern at Scout Space
Achieving precise and safe operations in space hinges on a transformative concept: digitizing the orbital environment. By distilling Earth's cluttered orbit into a comprehensive database of well-understood space objects, a digital map of space empowers safe space exploration. Beyond simply locating objects, a detailed characterization of these objects and their potential interactions represents a critical step towards creating this paradigm. This enhanced understanding will make space accessible and secure for exploration, enabling all stakeholders—from governments to private companies—to navigate and operate safely in this vital frontier.
Motivation
Capturing detailed 3D models of previously unknown objects is a key step to understanding and mapping the domain. Currently, visual observation informs space operations from docking maneuvers, to repair assessments and debris removal. However, the ability to not just see but to thoroughly analyze and manipulate objects in orbit remains a challenge. By converting inspection imagery into detailed 3D models, the data becomes actionable. This doesn't just enhance our ability to see objects in space—it empowers operators to interact with them more effectively, setting a new standard for space domain awareness and operational planning.
Scout has been collaborating with Noblis, a nonprofit science, technology, and strategy organization which is also an investor in Scout Space, to test and develop a 3D reconstruction algorithm that can run on-orbit via Scout’s Sparrow sensing platform. This collaboration leverages Noblis' expertise in advanced computer vision and space systems to enhance Scout's on-orbit processing capabilities. Our algorithm has been demonstrated on the imagery of a JAXA rocket body captured by Astroscale’s ADRAS-J vehicle. Figure 1 below shows the 3D model generated by this algorithm.
Figure 1: Scout’s reconstructed 3D model from 101 images from various angles of a rocket stage observed during Astroscale’s ADRAS-J inspection orbit.
Methodology
The 3D model reconstruction algorithm of choice is called Gaussian splatting. Gaussian splatting has emerged as a hallmark of the computer vision field, achieving unmatched quality with regards to its speed and low computational costs – a critical advantage for resource-constrained space systems. In simple terms, Gaussian splatting converts multiple images of an environment into a 3D model of it.
This method constructs a unique kind of 3D model; a lightweight grid of points in 3D space forms the geometry of the scene, while rich visual information is encoded in Gaussian distributions centered at each point. This structure allows complex visual features to be rendered efficiently, and avoids the problem of hallucinated visual detail that is seen in many diffusion-based models. Large datasets of images can be condensed into a lightweight model to be downlinked from space, offering significant financial savings when considering the costly per-minute rate of data transfer. By greatly reducing overall filesize, 3D reconstruction lowers the cost for actionable visualization of space objects.
Figure 2: Chart showing the relationship between the size of the image subsets of a 360-image set and the size of the resulting Gaussian splat model. In practice, at least 60 images per set is recommended, which is represented here by a 96% reduction in size!
Gaussian splatting synthesizes a model of the entire environment shown in a sequence of pictures, but for our purpose we want to model just one object. We want to isolate the object (likely a satellite, rocket stage, or orbital debris) from its celestial backdrop. Constructing a singular object offers a client targeted visualization for prospective docking, repair, or debris analysis in a more convenient format than swaths of images. Segmented object modeling also enables pre-mission planning, as simulation software can interact with these reconstructed shapes to rehearse complex operations before they happen.
Figure 3: A simulated set of robotic arms manipulating a pair of scissors generated by Gaussian splatting, one of many such experiments in a growing field of Gaussian-driven robotic manipulation. Gif courtesy of UC Berkeley
To create reconstructions of single objects, we use masks to pick the desired object out of each frame. The mask image is a binary representation of which pixels in the image contain the desired segment. The object is set to pure white, and the rest of the image is black. These masks, generated amongst the whole image set, will inform the geometry of our model.
With illumination dominated by the sun, lighting in space tends towards extremes. This volatility disrupts conventional masking algorithms, prompting Scout’s extensive research collaboration with Noblis into a more resilient masking technique. Traditional image processing methods enforce various thresholds on the image data to bring out things like edges, noisy unlit elements, and a vague first impression of the object. This classical approach can be bolstered when used in tandem with image boundaries created by depth maps. These depth maps are created by machine learning models that assign each pixel in a source image a value based on its estimated distance from the camera. This layered procedure builds up even further in the form of applying large AI masking models like Meta’s SAM 2, a research development driving towards coherent masking across sequences of images. Masks can be derived from any single step of this process, but linking them together as preprocessing steps can lead to exceptional coherence in masks across an entire sequence of differently lit images.
Figure 4: The various stages of masking lead to a mask that nicely segments out the whole craft, even areas in harsh shadow
How might we derive a 3D shape from a bunch of 2D masks? The answer lies in locating where cameras would need to be placed in a 3D space to take the given pictures. By relocating these cameras in a process called Structure from Motion(SfM) we are able to ground a preliminary map of our 3D scene. Most Gaussian splatting pipelines incorporate COLMAP, the eminent SfM tool.
Figure 5: The red elements in this image depict the relocated cameras that COLMAP distilled from 21,000 images of a walk through Rome. Note that COLMAP creates some initial geometry of the scene, but that is not used in our application. Image courtesy of COLMAP.
As the project has progressed, we have been working to create an internal tool to replace COLMAP’s camera registration by leveraging Scout’s extensive suite of distance and attitude detection software. COLMAP excels at estimating camera positions throughout continuous sequences of dense environments, but the application in space is better suited for a different approach. An object floating in space is often the only object in the entire frame, so camera registrations only need to consider the relation between the target and the camera. In addition, COLMAP sometimes struggles with nonsequential sets of images, so it wouldn’t be able to utilize Scout’s proposed distributed network of space sensors. Most importantly, COLMAP culls images in a sequence that it cannot relocate a camera for, drastically affecting the quality of an eventual render. Scout already develops sophisticated targeting algorithms, so these could be leveraged in an internal camera registration tool to outperform standard approaches.
Figure 6: The two different camera registration paradigms, with the red elements representing identified camera locations from the image sequence. The novel approach preserves all the images in the set, leading to more stable render quality.
The combination of image masks and relocated cameras gives us enough information to identify the geometry of the object. By projecting rays in the shape of each mask from its camera position, we can construct a cloud of points at their intersection. This point cloud depicts the initial geometry of the object.
Figure 7: An animated demonstration of how the initial point cloud structure is made from the image masks. Cameras are located around the scene, masks are projected out as rays from those cameras, and the intersected hull is identified as the geometry of the object.
Each point in this point cloud is then trained through the core Gaussian splatting stage to hold a 3D distribution of the observed visual data at that point from multiple perspectives. That distribution can be compressed onto the point in the final rendering step, hence the name ‘splatting.’
Conclusion
Scout’s 3D reconstruction application synthesizes several quickly advancing fields, positioning Scout’s well-established tracking capabilities as a distinct advantage for ongoing and future advancements in this domain. As a whole, the project provides an elegant solution to the unique challenges of space operations. By building upon our expertise in rendezvous proximity operations (RPO) payloads and object tracking software, we've created a framework that not only solves current visualization needs but anticipates future demands in simulating orbital operations. As these reconstruction technologies continue to evolve, Scout remains uniquely positioned to innovate towards robust operation planning and space-domain awareness.