Setup of the gesture based approach. (left) Kinect device and computer screen displaying the graphical user interface and interaction panel. (right) user performing a grip gesture.
(Left) Graphical user interface for gesture-based 2D/3D initialization. (right) interaction panel indicating control mode, and cursors indicating the current gesture classification (unknown, release, grip). The four primary panes control orientation and translation. The bottom left pane and left sidebar indicate the operator's hand location in the Kinect's workspace. The top left pane toggles between the x-ray images.
Calculation of pose update to the 3D data for a user interaction. W is the world coordinate frame, C and C′ are the coordinate frames of the reference x-ray at its current and virtual new positions, S and S′ are the coordinate frames of the 3D data at its current and actual new positions. Dashed line marks the unknown transformation.
Data acquisition steps shown from left to right: Original depth map where lighter colors are closer and darker colors are farther away. The joint positions are visualized as a skeleton overlaid onto the depth map; ROI surrounding the right hand; and downsampled and normalized depth map in ROI.
Variability of hand sizes used for training the classifier.
(a) The user attempts to mimic the volume-tool pose defined in the virtual world to the patient-tool pose in the physical world. That is, positioning the tool in the physical world such that . When this happens, the volume and patient coordinate frames coincide, and the unknown can be computed by composing the known with the tracked . (b) Coordinate systems involved in the initialization, solid lines denote known transformations, dashed denote transformations to be determined.
Graphical user interfaces for (a) pose planning in virtual world and (b) pose mimicking in physical world.
Reference data sets used in this study. The x-ray images for the ISI and Vienna data sets were provided in the anterior-posterior and lateral views. The Ljubljana data set provided 18 evenly spaced x-ray images around the spinal cord, and we selected two images (000 and 006) in our study. Original x-ray images for this data set had low contrast, so they were thresholded and enhanced to provide better visualization. The right two columns show the corresponding 2D rendering of the CT and MR images at their ground truth positions. They show the differences between x-ray images and rendering of the 3D data. The physical dimensions for x-rays, CT, and MR are shown underneath the images for each data set.
(a) Transformations involved in the validation of AR-based initialization approach. The tracker' coordinate system is the common/world coordinate system used by the reference data set. (b) Definition of reference transformation. The transformation between the patient, bottom, and patient DRF, top, coordinate systems was used as the reference for our experiments.
Experimental results for gesture-based initialization. (left column) distribution of mTRE per operator, and (right column) distribution of interaction time per operator.
Initial versus final mTRE for the gesture-based method, diagonal marks the no-change line. Note that the initial mTREs for the Vienna data set (circle marker symbol) are larger due to the “lever” effect associated with rotating a larger object.
Experimental results for interactive AR-based initialization. (left column) distribution of mTRE per operator, and (right column) distribution of interaction time per operator.
Relationship between instant and interactive initialization: instant mTRE versus interactive mTRE (top), and instant mTRE versus interaction time (bottom).
Clinical setup for ACL reconstruction surgery, and a lateral x-ray (not from same procedure). X-ray shows knee flexed at about 120°. Preoperative diagnostic MR is acquired at full extension, 0° between femur and tibia.
Experimental results for instant coarse initialization (for each reference data set, the results were summarized from all participants and both x-ray/CT and x-ray/MR experiments).
Consolidated results for interactive AR-based initialization (results were summarized according to different categories).
Article metrics loading...
Full text loading...