| # Documentation | |
| ## Python scripts | |
| These files are for our monocular 3D Tracking pipeline: | |
| `main.py` Execute 3D photo inpainting | |
| `mesh.py` Functions about context-aware depth inpainting | |
| `mesh_tools.py` Some common functions used in `mesh.py` | |
| `utils.py` Some common functions used in image preprocessing, data loading | |
| `networks.py` Network architectures of inpainting model | |
| MiDaS/ | |
| `run.py` Execute depth estimation | |
| `monodepth_net.py` Network architecture of depth estimation model | |
| `MiDaS_utils.py` Some common functions in depth estimation | |
| ## Configuration | |
| ```bash | |
| argument.yml | |
| ``` | |
| - `depth_edge_model_ckpt: checkpoints/EdgeModel.pth` | |
| - Pretrained model of depth-edge inpainting | |
| - `depth_feat_model_ckpt: checkpoints/DepthModel.pth` | |
| - Pretrained model of depth inpainting | |
| - `rgb_feat_model_ckpt: checkpoints/ColorModel.pth` | |
| - Pretrained model of color inpainting | |
| - `MiDaS_model_ckpt: MiDaS/model.pt` | |
| - Pretrained model of depth estimation | |
| - `use_boostmonodepth: True` | |
| - Use [BoostMonocularDepth](https://github.com/compphoto/BoostingMonocularDepth) to get sharper monocular depth estimation | |
| - `fps: 40` | |
| - Frame per second of output rendered video | |
| - `num_frames: 240` | |
| - Total number of frames in output rendered video | |
| - `x_shift_range: [-0.03, -0.03, -0.03]` | |
| - The translations on x-axis of output rendered videos. | |
| - This parameter is a list. Each element corresponds to a specific camera motion. | |
| - `y_shift_range: [-0.00, -0.00, -0.03]` | |
| - The translations on y-axis of output rendered videos. | |
| - This parameter is a list. Each element corresponds to a specific camera motion. | |
| - `z_shift_range: [-0.07, -0.07, -0.07]` | |
| - The translations on z-axis of output rendered videos. | |
| - This parameter is a list. Each element corresponds to a specific camera motion. | |
| - `traj_types: ['straight-line', 'circle', 'circle']` | |
| - The type of camera trajectory. | |
| - This parameter is a list. | |
| - Currently, we only privode `straight-line` and `circle`. | |
| - `video_postfix: ['zoom-in', 'swing', 'circle']` | |
| - The postfix of video. | |
| - This parameter is a list. | |
| - Note that the number of elements in `x_shift_range`, `y_shift_range`, `z_shift_range`, `traj_types` and `video_postfix` should be equal. | |
| - `specific: '' ` | |
| - The specific image name, use this to specify the image to be executed. By default, all the image in the folder will be executed. | |
| - `longer_side_len: 960` | |
| - The length of larger dimension in output resolution. | |
| - `src_folder: image` | |
| - Input image directory. | |
| - `depth_folder: depth` | |
| - Estimated depth directory. | |
| - `mesh_folder: mesh` | |
| - Output 3-D mesh directory. | |
| - `video_folder: video` | |
| - Output rendered video directory | |
| - `load_ply: False` | |
| - Action to load existed mesh (.ply) file | |
| - `save_ply: True` | |
| - Action to store the output mesh (.ply) file | |
| - Disable this option `save_ply: False` to reduce the computational time. | |
| - `inference_video: True` | |
| - Action to rendered the output video | |
| - `gpu_ids: 0` | |
| - The ID of working GPU. Leave it blank or negative to use CPU. | |
| - `offscreen_rendering: True` | |
| - If you're executing the process in a remote server (via ssh), please switch on this flag. | |
| - Sometimes, using off-screen rendering result in longer execution time. | |
| - `img_format: '.jpg'` | |
| - Input image format. | |
| - `depth_format: '.npy'` | |
| - Input depth (disparity) format. Use NumPy array file as default. | |
| - If the user wants to edit the depth (disparity) map manually, we provide `.png` format depth (disparity) map. | |
| - Remember to switch this parameter from `.npy` to `.png` when using depth (disparity) map with `.png` format. | |
| - `require_midas: True` | |
| - Set it to `True` if the user wants to use depth map estimated by `MiDaS`. | |
| - Set it to `False` if the user wants to use manually edited depth map. | |
| - If the user wants to edit the depth (disparity) map manually, we provide `.png` format depth (disparity) map. | |
| - Remember to switch this parameter from `True` to `False` when using manually edited depth map. | |
| - `depth_threshold: 0.04` | |
| - A threshold in disparity, adjacent two pixels are discontinuity pixels | |
| if the difference between them excceed this number. | |
| - `ext_edge_threshold: 0.002` | |
| - The threshold to define inpainted depth edge. A pixel in inpainted edge | |
| map belongs to extended depth edge if the value of that pixel exceeds this number, | |
| - `sparse_iter: 5` | |
| - Total iteration numbers of bilateral median filter | |
| - `filter_size: [7, 7, 5, 5, 5]` | |
| - Window size of bilateral median filter in each iteration. | |
| - `sigma_s: 4.0` | |
| - Intensity term of bilateral median filter | |
| - `sigma_r: 0.5` | |
| - Spatial term of bilateral median filter | |
| - `redundant_number: 12` | |
| - The number defines short segments. If a depth edge is shorter than this number, | |
| it is a short segment and removed. | |
| - `background_thickness: 70` | |
| - The thickness of synthesis area. | |
| - `context_thickness: 140` | |
| - The thickness of context area. | |
| - `background_thickness_2: 70` | |
| - The thickness of synthesis area when inpaint second time. | |
| - `context_thickness_2: 70` | |
| - The thickness of context area when inpaint second time. | |
| - `discount_factor: 1.00` | |
| - `log_depth: True` | |
| - The scale of depth inpainting. If true, performing inpainting in log scale. | |
| Otherwise, performing in linear scale. | |
| - `largest_size: 512` | |
| - The largest size of inpainted image patch. | |
| - `depth_edge_dilate: 10` | |
| - The thickness of dilated synthesis area. | |
| - `depth_edge_dilate_2: 5` | |
| - The thickness of dilated synthesis area when inpaint second time. | |
| - `extrapolate_border: True` | |
| - Action to extrapolate out-side the border. | |
| - `extrapolation_thickness: 60` | |
| - The thickness of extrapolated area. | |
| - `repeat_inpaint_edge: True` | |
| - Action to apply depth edge inpainting model repeatedly. Sometimes inpainting depth | |
| edge once results in short inpinated edge, apply depth edge inpainting repeatedly | |
| could help you prolong the inpainted depth edge. | |
| - `crop_border: [0.03, 0.03, 0.05, 0.03]` | |
| - The fraction of pixels to crop out around the borders `[top, left, bottom, right]`. | |
| - `anti_flickering: True` | |
| - Action to avoid flickering effect in the output video. | |
| - This may result in longer computational time in rendering phase. | |