Abstract

Light-detection and ranging (LiDAR) is being increasingly deployed for consumer imaging across handheld, wearable and robotic applications 1 , 2 , 3 , 4 . These sensors measure the time-of-flight of light at picosecond resolution, which could enable them to image objects hidden from their field of view. Although such non-line-of-sight (NLOS) imaging capabilities have been shown on research-grade LiDAR devices, they remain challenging to achieve on consumer devices due to poor signal quality resulting from low laser power, low spatial resolution, and object and camera motion. Here we propose a multi-frame fusion strategy to overcome these challenges and demonstrate NLOS imaging on consumer LiDAR. We introduce the motion-induced aperture sampling model to unify the effects of object shape, object motion and camera motion under a single measurement model. Using this model, we demonstrate several NLOS capabilities on a smartphone-grade LiDAR: (1) three-dimensional reconstruction; (2) single- and multi-object tracking; and (3) camera localization using hidden objects. Previously, NLOS imaging capabilities were restricted to bulky and expensive research-grade hardware that requires extensive set-up and calibration. Our results represent a shift towards plug-and-play NLOS imaging, where anyone can image hidden objects with off-the-shelf hardware (for less than US$100) and no additional set-up. We believe democratization of such capabilities will advance consumer applications of NLOS imaging.

Main

Today, most consumer imaging systems—handheld, wearable or robotic—can only see objects within the direct line of sight of the camera. But what if these everyday cameras could also see beyond the line of sight (Fig. 1a,b)? Augmented-reality systems would be able to robustly map the user’s body pose for seamless integration into the virtual world. Warehouse robots would be able to navigate fast-paced and unstructured environments while avoiding collisions around the corner. Home robots would remain oriented in textureless spaces by using temporarily hidden objects to localize themselves. Motivated by such use cases, we revisit the non-line-of-sight (NLOS) imaging problem through the lens of consumer mobile imaging.

a, NLOS imaging is possible by turning nearby diffuse surfaces into virtual mirrors that reveal hidden objects; however, NLOS imaging with consumer LiDAR devices is challenging due to low SNR, the tradeoff between virtual aperture size and sampling density, and joint object and camera motion. We address these challenges using multi-frame fusion. b, We demonstrate three consumer NLOS capabilities—three-dimensional reconstruction, tracking and camera localization—on smartphone-grade LiDAR. The insets of the application figures shows the limitation of only relying on line-of-sight signals. ICP, iterative closest point; GT, ground truth; FOV, field of view.

Several approaches have been proposed for NLOS imaging 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , but the most promising for mobile deployment is based on time-of-flight (TOF) imaging 16 , 17 , 18 , 19 , 20 , which extracts faint signals from hidden objects by analysing the travel time of light. Although early TOF systems were bulky and expensive 17 , 21 , 22 , recent advances in single-photon light-detection and ranging (LiDAR) 23 , 24 have made such sensors ubiquitous. Now, LiDARs are integrated into consumer mobile devices such as smartphones 2 , 4 , augmented-reality headsets 1 and autonomous vehicles 3 . Although most LiDARs process raw TOF measurements into three-dimensional (3D) point clouds of visible surfaces, the raw data also contain indirect reflections that, in principle, encode information about hidden objects.

But can consumer LiDARs be used to image hidden objects? Extensive work has been done for LiDAR-based NLOS imaging 18 , 19 , 20 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 . However, these methods are designed on the basis of idealized conditions of laboratory-grade LiDAR set-ups that do not reflect several constraints of consumer mobile sensing (Table 1). First, low laser power (due to eye safety constraints) and short exposure times (due to dynamic scenes) result in measurements with a low signal-to-noise ratio (SNR). Second, constraints on bandwidth, power and chip area limit spatial resolution, forcing a trade-off between aperture sampling density and total aperture size. Finally, the joint motion of the object and camera results in motion blur if improperly handled. Intuitively, these constraints result in lower-quality measurements than those typically assumed in past work, motivating the need for algorithms specifically for the consumer NLOS regime.

Here we address these three challenges and demonstrate NLOS imaging on smartphone-grade LiDAR. Although past works have primarily focused on single-shot reconstruction 19 , 33 —where the signal quality within each frame is sufficient for inference—we tackle the more practical case in which individual frames are insufficient in isolation, but collectively can inform a useful estimate of the hidden object. Our approach is inspired by burst photography 34 , which exploits viewpoint redundancy to improve SNR and synthetic aperture radar 35 , which exploits viewpoint diversity to improve the reconstructed resolution.

Our key technical contribution is the motion-induced aperture sampling (MAS) model, which enables multi-frame fusion by unifying the effects of object shape, object motion and camera motion. This model decouples the effects of motion and object shape on a frame measurement, which is conducive to efficient processing algorithms. Our work also uses the observation that many 3D vision tasks are constrained to solve for a single unknown variable: 3D scanning assumes a known camera pose and static scene geometry 36 , 37 , 38 ; object tracking assumes a known object shape is available as a template 39 , 40 ; and camera localization assumes a known, static environment 41 , 42 . This problem formulation of solving for one of three unknowns—object shape, object position or camera position—enables efficient and tractable inference using the MAS model.

Using these insights, we experimentally demonstrate three consumer NLOS imaging capabilities (Fig. 1c). First, we show 3D reconstruction of static hidden objects by exploiting the natural handheld camera motion (with known pose) in mobile systems. Second, we demonstrate 3D tracking of single and multiple hidden objects of known shape. Third, we show camera localization using known static hidden objects as visual cues, which is especially useful when standard visual odometry techniques struggle (for example, when imaging textureless or planar surfaces). We achieve each of these capabilities with real-time on-line processing, making it feasible for many practical applications.

To our knowledge, our results are the first such demonstrations of NLOS capabilities on widely available consumer mobile LiDARs. Previously, NLOS imaging set-ups were restricted to expensive and bulky hardware with complicated set-up and calibration procedures. Our work represents a step towards plug-and-play NLOS imaging—we show for the first time that anyone can image hidden objects with off-the-shelf hardware (for less than US$100) and no challenging calibration. With the advent of consumer LiDARs, we envision that our work will democratize access to NLOS imaging capabilities for practitioners across many disciplines—including robotics, computer vision, medicine, and user interface or user experience—and spur new applications of NLOS imaging.

Capturing NLOS signals with consumer LiDAR

In our experiments, we use a portable smartphone LiDAR system with approximately 100 pixels, each consisting of a co-located laser emitter and single-photon avalanche diode (SPAD) detector. Each SPAD pixel shares an optical axis with exactly one laser spot. Although the configuration is similar to the confocal capture set-up used by O’Toole et al. 18 , our set-up is integrated onto an existing smartphone-grade LiDAR. As all pixels and laser spots are simultaneously active, we assume that the hidden object of interest has strong retroreflective properties to retain the image formation model of confocal scanning set-ups 18 . However, we empirically show that our model works on diffuse objects as well. We assume object motion is a rigid-body translation (no rotations). The camera is handheld and free to move in an unstructured manner, but must be aimed towards the relay surface (that is, the virtual mirror) at z = 0 (Fig. 1a). We refer to the imaged area on the wall as the virtual aperture. The camera captures signals from the hidden object by measuring light that travels from the laser, to the virtual aperture, hidden object, virtual aperture and then the SPAD pixel. Each pixel measures picosecond-scale light transport by recording the distribution of light intensity as a function of time. From these space-time measurements, we infer properties of the hidden object.

Space-time impulse response

The space-time impulse response (STIR) is a continuous-valued function i(x,y,τ) that fully describes the hidden scene. The STIR contains the temporal response i(τ) at every sampled (x,y) location on the relay wall, assumed to be at z = 0. Past works have shown that the hidden scene can be estimated if the STIR is captured with sufficient SNR, spatio-temporal resolution and virtual aperture size 16 .

Consumer LiDARs, however, capture a noisy, low-spatial-resolution and cropped estimate of the STIR hati(x,yτ). As a result, a single frame is not informative enough to reconstruct the hidden scene; however, if the hidden scene is static, we can leverage the portability of the LiDAR system by moving the camera and scanning the relay surface (as shown in Fig. 1a). This camera motion results in increased synthetic aperture size and resolution due to further spatial sampling of the STIR. Using multiple frames also improves SNR by exploiting redundancy in measurements. Once sufficiently sampled, the STIR can be used to reconstruct the hidden scene using techniques that do not rely on uniform spatial sampling 17 , 20 . We show such a 3D reconstruction using filtered backprojection in Fig. 1b. We show the benefits of multi-frame fusion in Supplementary Fig. 11 and ablate the effect of pose noise in Supplementary Fig. 12.

Motion-induced aperture sampling model

Under the presence of object motion, the STIR will change over time. As a result, techniques such as filtered backprojection will fail due to motion blur. For mobile applications, we must model the joint effects of object shape, object motion and camera motion to accurately recover NLOS scenes.

At a given object position, the light-cone transform (LCT) states that the relationship between the object’s volumetric albedo ρ(x,y,z) and its continuous-valued STIR i(x,y,τ) can be expressed as a 3D convolution

(1)

where ℜ τ and ℜ z are non-linear resampling operators in time and space respectively, the * represents the 3D convolution operator, and h(x,y,v) is the three-dimensional point spread function (PSF). Due to the convolutional image formation model in LCT-transformed space, we will treat all measurements in the transformed space ℜ τ {i} = I(x,y,v). Without loss of information, we can treat I(x,y,v) as the STIR of the hidden scene.

Under a rigid-body object translation of Δ = (Δx,Δy,Δv), the STIR also translates by Δ due to the shift-invariance of the convolution operation. As a result, the STIR at time t can be expressed as

(2)

This property allows us to decompose the time-varying STIR into a time-independent canonical STIR I 0 (x,y,v), and a time-dependent object shift Δ = (Δx t ,Δy t ,Δv t ). A proof of this decomposition is provided in Supplementary Note 1.

The camera acts as a spatial sampling function, sampling the STIR at spatial locations determined by the camera pose and with fixed timing resolution. The measured spatio-temporal profile at time t, 2D pixel location u and temporal bin v can be expressed as

(3)

where P t is the camera extrinsic matrix at time t; K is the camera intrinsic matrix; x = [x, y, 0] T are spatial locations on the relay wall, x′= [x, 1] T is the homogeneous coordinates of x; and i(x, v) is the LCT-transformed transient measurement at pixel location u.

Equation (3) defines the MAS model. The second term encodes the time-independent object shape through the canonical STIR I 0 (x,y,v) and the time-dependent object position through Δ t . The first term models the effect of the camera pose P t as a spatial sampling of the continuous STIR function. In practice, the signal is also temporally quantized due to the sensor timing resolution, but we ignore it for conceptual simplicity. The combined effects of object shape, object motion and camera motion are summarized in Fig. 2: object shape defines a canonical STIR I 0 (x,y,v), object motion defines the translation Δ t of the canonical STIR, and camera pose P t determines the sampling of the translated STIR.

The space-time measurements can be expressed as the combined effect of object shape, object motion and camera pose using the convolutional property of the LCT. LCT aligns scene space and measurement space by transforming depth in scene space as v z = z 2 and time in measurement space as v τ = (cτ/2) 2 .

Single and multi-object 3D tracking

Jointly recovering object shape, object position and camera pose under the MAS model is a highly non-convex inverse problem that would require iterative solvers. Such iterative solvers, however, are computationally expensive and unsuitable for real-time execution on consumer platforms. We avoid the need for such solvers by using the observation that many applications require solving for one unknown variable at a time. In tracking, the object of interest often has known (or approximately known) shape, for example, when a line-of-sight object moves out of view 39 , 40 . In robotics, camera localization assumes that the environment is known when recovering the camera position 41 . Motivated by such applications, we outline a technique for real-time object tracking and camera localization based on the MAS model. We first focus on object tracking and later extend it to camera localization.

Non-line-of-sight tracking is challenging because of the low spatio-temporal resolution and low SNR of the measurements. As a result, we want to be able to (1) robustify our estimate of the object position by incorporating information from past measurements, and (2) model the uncertainty of our estimates on the basis of the quality of the measurements. We use a particle filter to achieve both of these goals.

Our goal is to model the posterior distribution P(Δ t |i t , Δ t – 1 ), that is, the probability of the current object position Δ t given the current measurement i t and previous object position Δ t – 1 . A particle filter allows us to represent arbitrarily complex distributions by exploiting the duality between a distribution and samples generated under it 41 , 43 . As a result, we can model the posterior by storing N random samples (that is, particles) from the distribution. The kth particle boldsymbolDelta in R d represents a guess of the object position at time t, where d = 3 because the state is the 3D position of a single object. The final posterior distribution can be estimated using kernel density estimation 44 .

The particle-filtering algorithm enables sequential processing through a decomposition of the posterior distribution into two terms: the motion model and the data likelihood

(4)

This decomposition enables straightforward estimation of the posterior through three steps: particle propagation, evaluation and resampling. First, each particle is propagated on the basis of the probabilistic motion model P(Δ t |Δ t –1 ). The choice of the motion model encodes a prior assumption as to how the object moves from one time step to the next (for example, constant velocity). Second, each particle is evaluated using a score function that defines the data likelihood term P(i t |Δ t ). This step assigns a weight to each particle by (1) rendering the measurement via the MAS model and (2) assigning a score on the basis of how similar the rendering is to the measurement. Finally, the particles are resampled using importance sampling. Intuitively, the resampling step probabilistically eliminates particles that are highly inconsistent with the measurement and introduces particles that are more consistent with the measurement. After this step, all particles have equal weight and collectively parametrize the posterior. The particle filter algorithm is summarized in Extended Data Fig. 1 and Supplementary Algorithm 1. We also extend this formulation to camera localization in the Methods .

Although our approach is inspired by the multi-frame fusion principle used in burst photography, it does not implement burst imaging in the conventional sense. Traditional burst pipelines rely on explicit frame alignment or registration followed by integration to improve the SNR. Such image-based alignment is difficult to apply in our context because each frame has a very low spatial resolution (~100 pixels). Rather than aligning frames directly, particles hypothesize object locations within a neighbourhood of the previous location. Another way to interpret this analogy is that burst imaging aligns in image space, whereas our approach aligns in state space.

We demonstrate single-object tracking in Fig. 1c. For these results, we omit plotting the first few frames, which correspond to the object localization stage. We use the aforementioned particle-filtering technique to estimate the object position. To output a single position estimate for each frame (rather than distribution), we compute the mean of the distribution by computing the mean of the particles. The trajectory is then plotted as the mean estimate over all frames. The predicted trajectory is shown in blue and the ground truth trajectory is in black; NLOS imaging is inherently ill-posed, and many object positions will be consistent with the measurements. A useful feature of particle filtering is that it can implicitly capture these regions of ambiguity through the posterior, and even break the ill-posedness to an extent by leveraging the motion prior. These tracking distributions can be visualized in Supplementary Video 1. We also compare our proposed tracking approach with approaches based on conventional backprojection techniques 45 , 46 in Supplementary Fig. 2 and show how our approach improves tracking of large objects in Supplementary Figs. 4, 5 and 13. We also show a handheld camera localization result in Supplementary Video 3 and Supplementary Fig. 7.

We can also perform multi-object tracking in a similar manner with a small modification to the particle-filtering formulation. If we want to track M objects, then the size of our state space would be d = 3 M and the particle filter would represent the joint probability distribution of all object positions

(5)

where Δ t ( m ) represents the position of the mth object at time t. The rendered measurement of the kth particle at time t would then be the linear super-position of all object renderings

(6)

where w t ,k is the dot product between hati t,k (m) /|hati t,k (m) | 2 and the measurement i t , and hati t,k (m) (bfu,v) is the rendered contribution of the mth object. We found that applying this weight normalization and ensuring that I 0 accounts for laser pulse width are crucial for performance. Furthermore, simply computing the mean of the posterior distribution does not work for multi-object tracking because the distribution is multi-modal (due to multiple objects). Instead, we cluster particles using k-means clustering, choose the cluster with the most amount of particles and then estimate the positions of the objects by computing the mean within that cluster. Although we do not account for object occlusions here, the model could easily account for them by modifying equation (3) to include a pixel-wise visibility term that models the visibility between pixel u and object m based on other objects.

We demonstrate multi-object tracking in Fig. 3. We also demonstrate NLOS hand tracking as an application enabled by our method in Supplementary Video 2. In this application, the user is wearing retroreflective gloves. As a result, the NLOS measurements are dominated by the two bright patches (corresponding to the user’s hands), effectively reducing the problem to tracking the two brightest points in the scene. The signal reflection from the torso will be much weaker in this case and therefore does not cause confusion. This set-up is analogous to motion-capture systems, which attach retroreflective markers to key points on the body to enable reliable signal detection. The probability distributions of the left- and right-hand positions are shown in green and blue contour maps, respectively. As the left hand is further from the virtual aperture than the right hand, it’s position becomes more uncertain due to a larger region of ambiguity.

a, We can detect and track multiple objects. In this result, we show tracking of one moving object on a translation stage and one static object. The first few frames are localizing the object, and are therefore omitted from the visualization. b, We demonstrate NLOS hand tracking as an application of multi-object tracking. The predicted right and left hand probability distributions are shown in blue and green respectively (Supplementary Video 2).

Discussion

We show that our proposed motion-induced aperture sampling can handle 6D (5D) camera motion when pose is known (unknown). Although we do not show results for 6D motion in camera localization, our model can handle the extra degree of motion as long as the object shape is not rotationally symmetric. The MAS model can also handle 4D object motion (3D translation and 1D roll rotation parallel to wall), but we only show the translation case. Future work could handle non-rigid-body motion by modelling a point-wise deformation field 47 , 48 of the canonical STIR. Handling object pitch and yaw rotation would require modelling visibility of points that come in and out of view of the relay surface.

We are able to handle challenges related to SNR and reconstruction resolution in tracking and camera localization by leveraging multi-frame fusion. Specifically, we leverage particle filtering to improve SNR by exploiting motion priors and handle resolution limits by explicitly modelling regions of ambiguity using probability distributions. One limitation of our work is that we do not model much of the complexity of the real world, such as complex reflectance properties of natural objects (for example, inhomogeneous bidirectional reflectance distribution functions), the viewing angle of the camera with respect to the relay surface, and sensor non-idealities (for example, pulse width). As a result, it is challenging to derive a score function that can robustly compare the rendered and actual measurements. In particular, we notice larger regions of ambiguity when dealing with larger objects. Future work could overcome this limitation by leveraging machine learning to learn a score function, rather than relying on handcrafted score functions.

Our model assumed that most measured light resulted from retroreflective reflections. Experimentally, these conditions held because we used objects covered in retroreflective materials. We also demonstrate that our model handles diffuse objects well in practice, as shown in Fig. 4 and Supplementary Video 1. However, the signal quality is worse due to poorer SNR and stronger reflections from non-confocal paths.

Although the MAS model we propose was derived assuming retroreflective object reflectance, we find empirically that the model can handle diffuse objects. We show that we are still able to reconstruct an object’s three-dimensional shape and motion in real-time (Supplementary Video 3). The results, however, are inherently worse than those obtained with retroreflective objects due to the weaker signal (due to r 4 falloff) and light contributions from non-confocal paths.

Here we assumed that two out of object shape, object motion (static or dynamic) and camera pose are known. Future work could extend this assumption to directly solve for all three unknowns, similar to the simultaneous localization and mapping (SLAM) formulation 49 . However, solving for object and camera motion jointly can be challenging due to degenerate cases, for example, if the object and camera translate the same amount parallel to the x–y plane. In such cases, only the relative motion of the object and camera can be recovered.

Although our work is focused on non-line-of-sight imaging, we believe that there are other practical uses of multi-bounce signals in consumer LiDARs. One such example is 3D mapping in constrained settings in which the number of pixels is limited (for example, minimalist sensing 50 ). In such cases, 3D mapping is challenging because conventional LiDAR maps a single scene point to a single point in the measurement. When fewer pixels are available, the measurement of the 3D shape will be aliased due to low spatial sampling rate. On the other hand, the third-bounce measurements of an object results in opportunistic blurring of the object shape with a PSF kernel h(x,y,v). Sampling this blurred signal is a form of anti-aliasing that can be useful for recovering the rough shape of an object. This idea is further discussed in Supplementary Note 3.2.

Historically, NLOS imaging has required expensive equipment with challenging calibration procedures. However, with the low-cost commercial availability of single-photon LiDAR sensors, we envision a new era of plug-and-play NLOS imaging. Instead of requiring tens of thousands of dollars and hours of calibration, plug-and-play NLOS imaging enables NLOS capabilities for less than US$100, and with seconds of calibration. In Supplementary Figs. 3 and 6 we show that an off-the-shelf ST VL53L8CX SPAD can be used for NLOS tracking and camera localization with no physical calibration or additional hardware needed. The code required to run such demonstrations will be publicly released. We hope that such plug-and-play capabilities will democratize access to NLOS imaging and spur further research applications (Fig. 5) along this direction.

For the bouncing ball example, we omit the last few frames (as the ball hits the ground) because the signal is weaker as the ball moves away from the virtual aperture.

Methods

Camera localization using NLOS objects

We can also use signals from hidden objects to localize the camera, especially when line-of-sight surfaces do not contain useful visual features (for example, planar white surfaces). Conventional visual odometry techniques (for example, structure from motion (SfM) 51 , iterative closest point (ICP) 52 ) would fail in such scenarios because of limited geometric and texture variation; however, if we can image outside the camera’s line-of-sight, we can use nearby objects as landmarks.

The goal of camera localization is to recover the position of the camera in a defined world coordinates. As the hidden object serves as the landmark, we define the origin of this coordinate system to be at the same (x,y) location as the object, with z = 0 being the wall plane. In the case in which we are imaging a planar surface, point cloud measurements from a LiDAR would constrain the 6D camera pose up to three degrees of ambiguity: 2D translation parallel to the wall and roll rotation parallel to the wall. Assuming no roll rotation parallel to the wall, only the (x,y) translation component of the pose P t is unknown. Mathematically, we could express the world coordinates of a point x on the wall imaged by pixel u as

(7)

Here, P −1 is the unknown cam2world matrix that we can decompose into rotation R and translation t components. The camera coordinates of the imaged point x c = K −1 u is known from the LiDAR depth and the camera intrinsics K. If there is no roll rotation with respect to the wall, then R can also be estimated via plane fitting. The z co-ordinate of the camera can then deterministically be computed from the fitted point cloud hatbfx c. Only the (x,y) component of the cam2world translation vector t = [x c , t ,y c , t ,z c , t ,1] T is unknown.

We can estimate this translation component by using the NLOS signal. The true point cloud is x w = hatbfx c + [t x ,t y 0] T . Using equation (3), we can render what the measurement looks like under different translations [t x ,t y ] T , assuming the object shape (that is, the canonical STIR) is known. This problem formulation naturally allow us to integrate the camera localization problem into a particle filtering framework similar to the object tracking case. The camera localization algorithm is summarized in Algorithm 2 in the Supplementary Information. We show quantitative results in Fig. 1d and qualitative results with unstructured handheld motion in Supplementary Fig. 7 and Supplementary Video 3.

Pre-computing canonical STIR and rendering as an indexing operation

In equation (3), we showed that the object shape can be fully described by the continuous STIR of the object. The relationship between the object shape and this canonical STIR is determined by the PSF kernel

(8)

which parametrizes a parabola in LCT-transformed space-time. As a result, the canonical STIR of the object can be approximated as a sum of parabolas

(9)

where mathcalP is the point cloud defining the object shape, p = (p x ,p y ,p z ), p v = p z 2 and ρ p is the albedo at p. We normalize the point cloud such that the centroid of the points is defined to be at the origin. For point-like objects, we set ρ p = 1. For extended objects, we set ρ p = exp(–aδx 2 + bδy 2 ), where δx = x – p x , δy = y – p y , and a and b are hyperparameters. This position-dependent scale factor models the fact that points on the object closer to the synthetic aperture would have stronger signal than those further away. Empirically, we observe that using such radiometric scaling improves tracking accuracy for non-point objects.

Under object translation, the object shape as viewed by the relay surface is time-invariant. As a result, we can pre-compute the canonical measurement I 0 as a voxel cube hatI 0 in R^N x * N y * N v, where N x , N y and N v are the number of points sampled along the x, y and v direction of the canonical STIR. Once this STIR is pre-computed at pre-determined spatial-temporal points, any measurement can be efficiently rendered by simply indexing into the correct space-time points using equation (3).

Details on particle filtering

We use 1,000 particles to parametrize the posterior distribution. The three main steps of particle filtering are particle propagation, evaluation and resampling. Particle propagation requires specification of a motion model or prior, which we define as

where I is the identity matrix, mathcalN(μ, Σ) is a normal distribution with mean μ and co-variance Σ, and r is a measure of the maximum expected radius that the object is expected to travel between consecutive frames. The motion prior can also be based on second-order information (for example, velocity, acceleration), but we find that a proximity prior with r = 5 cm already works well empirically at 30 Hz capture. The particle evaluation step requires comparison of the rendered measurement hati t in R^n x * n y * n v of the kth particle with the actual measurement i t in R^n x * n y * n v. We use a normalized dot product score function

(10)

where score k is the unnormalized score of the kth particle, and η is a hyperparameter governing the relative strength of the motion prior and the data likelihood. The score function can be interpreted as a way of performing tracking via template matching, where the template is based on the object shape and MAS model for NLOS imaging. Intuitively, a higher value of η increases the separation between lower- and higher-valued particles, indicating that the algorithm should rely on the data likelihood more than the motion prior. When η is set to be lower, the algorithm will not favour one particle significantly more than the other, reducing the impact of the data likelihood. This case is useful when individual measurements are less reliable or noisier. Although η could be defined adaptively (for example, based on per-frame SNR), we set η to be fixed within an experiment. Once the score is computed for all particles, the score is normalized to produce a valid probability distribution. Finally, we use residual resampling 53 to maintain a balance between low-variance sampling and particle diversity. The pseudocode algorithm for object tracking and camera localization are provided in Algorithms 1 and 2, respectively, in the Supplementary Information. Ablations of the particle filtering parameters are provided in Supplementary Figs. 8–10.