High-quality binary fringe patterns generation by combining optimization on global and local similarity

Background

The recently proposed optimized dithering techniques can improve measurement quality. However, the objective function in these optimization methods just qualifies the global similarity of the pattern while the global optimization methods ignore the influences of local structure.

Method

In order to get high quality binary dithered patterns, this paper presents an algorithm which combines optimization on global and local similarity and implements it in the whole-fringe optimization.

Results

Both the simulation and experimental results demonstrate that the proposed algorithm can achieve binary fringe patterns with higher phase quality and less intensity difference. The fringe patterns based on the proposed method can get high quality measurement results.

Conclusion

The proposed method can get high quality binary dithered patterns under different defocusing levels by combining global similarity and local structure similarity. The proposed method can improve the robustness to the amounts of defocusing.

Digital fringe projection (DFP) techniques based on sinusoidal fringe patterns have been widely used for high-quality 3D shape measurement due to their flexibility and speed [1]. However, there are major limitations of the conventional DFP technique: high speed measurement (i.e., typically 120 Hz) and projection nonlinearity, which make it difficult to be applied to high-speed measurement [2].

To overcome these limitations, two categories of methods are proposed: the 1-D pulse width modulation methods [3, 4] and the 2-D area modulation methods [5, 6]. The former separates the fundamental frequency from high frequency harmonics by either shifting or eliminating high frequency harmonics. This method is successful to meet the requirement of high speed measurement, but this technique also has the defects: (1) when the fringe stripes are wide, the improvements have been limited [7], (2) because the pulse width modulation (PWM) technique is one-dimensional pattern, it cannot fully take advantage of the two-dimensional binary defocusing technique [8]. Xian and Su [9] propose the area-modulation technique, which can generate high-quality sinusoidal fringe patterns for precise micro manufacturing. In 2012, Lohry and Zhang [10] propose a technique to approximate the triangular waveform by modifying 2 × 2 pixels so that it can get better measurement results. However, if the fringe stripes are wide, it is difficult to achieve high-quality patterns.

Dithering techniques have been extensively used to produce high quality fringe pattern that can approximate the ideal sinusoidal patterns more closely after defocusing [11, 12]. The superiority of dithering techniques is that it operates a two-dimensional process that can manipulate the noise more flexibly and imitate sinusoidal fringes, especially it can get high quality fringe pattern for large period. Over the past years, some researchers have developed the dithering techniques to make significant improvements to fringe pattern quality. These studies include: Bayer method [13], error diffusion method [14] and genetic method [15]. The quality of the pattern based on optimization method is higher than that of other methods, and the genetic method improves the phase drastically. However the dithering techniques are originally developed for global similarity [16, 17].

In order to improve local details and the quality of fringe patterns, a lot of research has been carried out to optimize the dithered fringe patterns for phase-shifting profilometry. The optimization can be carried out in either intensity or phase domain. The former approach [18, 19] tries to minimize the error between the defocused halftone pattern and a sinusoidal fringe pattern while the latter approach [20] tries to minimize the phase error achieved with the defocused halftone patterns. Since the quality of measurement is determined by the phase error, the phase-based optimization tends to optimize the measurement quality directly while the intensity-based optimization does not. However, the performance of current phase-based optimization method is more sensitive to the extent of defocusing which may not be controlled precisely in practical situations [21]. The intensity-based method is robust to the amounts of defocusing. Whereas, these optimization methods just focus the global intensity similarity while ignore the influences of local structure. The intensity-based optimization and phase-based optimization are always implemented in a small patch of the fringe pattern and the best patch is tiled through symmetry and periodicity. In the process of optimization, manual selection of the best patch is very tedious and time-consuming.

This paper proposes a new optimization framework to improve the fringe pattern quality. In order to make sure global and local similarity between the ideal pattern and the defocused dithering pattern. We propose an objective function which weighted combines normalized mean squared error and structural similarity index measure. The global similarity is presented by normalized mean squared error and the local similarity is presented by structural similarity index measure. The goal of optimization is to minimize the object function between the defocused binary pattern and its corresponding ideal sinusoidal pattern.

The remainder of this paper is organized as follows. Section Method, we briefly review the principle of three-step phase-shifting algorithm, the principle of the object function and the whole fringe optimization framework. Section Results and discussion, we present the experimental results. Section Conclusion provides the conclusion of this paper.

Three-step phase-shifting algorithm

In this paper, three-step phase-shifting algorithm with a phase shift 2π/3is applied. Because it need the least formulas for its simplicity and speed. The sinusoidal light intensity can be described as:

WhereA(x, y) is the average intensity. B(x, y) is the intensity modulation. φ(x, y) is the phase needed to be calculated. Based on the Eqs.(1–3), the φ(x, y) can be gotten.

Notably, the environmental noise can be eliminated by subtraction of different patterns. The solved phase from Eq. (4) is wrapped in range (−π, π]. After φ(x, y) is unwrapped, the absolute phase can be obtained.

Objection function

Dithering optimization technique has been researched in reducing its overall phase error. The objective function of all these optimization techniques is to get the best fit of the binary patterns to the corresponding ideal sinusoidal pattern [14]. The optimization process can be described as the norm function such as norm Frobenius function:

Where ‖·‖ represents the norm Frobenius. I(x, y) is the ideal sinusoidal intensity pattern. G(x, y) represents a 2D Gaussian function. B(x, y) is the 2D binary pattern. * represents convolution. The optimization function evaluates the global intensity similarity between the dithering pattern and its corresponding ideal pattern, but it just considers the global similarity without focusing on local structure similarity. In addition, the global optimization is a very time-consuming process and it is difficult to solve the problem by using the objective function because of the NP problem [18].

The local detailed structure is one of the important information resources in an image. It contains particular high-frequency components. In order to contain the local similarity in the objective function, we propose a weighted synthetic function (WSF). The objective function combines two parts: a global intensity part and a local structure part. The global intensity measurement is based on the normalized mean squared error (NMSE). The local structure measurement is based on the structural similarity index measure (SSIM). The WSF can be expressed as follows:

Where NMSE(I, I _d ) measures the global similarity and SSIM(I, I _d ) measures the local similarity, respectively. WSF gets the synthetic error between the ideal pattern I and the dithering pattern I _d . The ω _n and ω _s are the weight factors (ω _n  + ω _s  = 1). Because ω _n is the coefficient of the normalized mean squared error, it represents the weight of global similarity, such as period, peak, trough and symmetric axis and so on. While ω _s is the coefficient of structural similarity index measure which represents the weight of local similarity such as local luminance, contrast and structure and so on. In this paper, ω _n and ω _s are empirically set to be 0.7 and 0.3 respectively.

First a global intensity similarity is introduced into the WSF expression. The NMSE(I, I _d ) can evaluate the global intensity similarity between the dithering pattern and the corresponding ideal pattern. The NMSE(I, I _d ) can be shown as:

Where H and W respectively represent height and width of the pattern. The NMSE(I, I _d ) method reflects the whole fringe pattern intensity statistical errors. Thus, the smaller the NMSE(I, I _d ) is, the better the whole pattern quality becomes. However NMSE(I, I _d ) only focuses on the global similarity, so SSIM(I, I _d ) is introduced as an optimization term to evaluate the local structure similarity between the dithering pattern and the corresponding ideal pattern. The whole fringe pattern is divided into P elements by the neighborhood window. SSIM(I, I _d ) is calculated in the neighborhood window and the window size is designed based on the whole pattern. For example, in this paper, the size of captured image is 768*1024. Because the vertical fringe pattern is used, the window size is designed according to the width of fringe pattern. 24 pixels are neglected from the 1024 pixels and 1000 pixels are kept. For the whole fringe pattern, it can be divided into 20 parts where every part contains 50 pixels. Every part is analyzed by using luminance, contrast and structure. At last every optimized section is connected directly without complex computing because we optimize the fringe pattern in a whole pattern. In the local structure, the comparison between the ideal pattern and dithering pattern focuses on three parts: luminance, contrast and structure. First the local structure function (LSF) should be generated and the structural intensity error is calculated by comparing the LSF between I(x, y) andI _d (x, y).

The calculation process of LSF(x, y) is shown as following and we set I(x, y) as an example:

Luminance: Luminance function can be designed as l(x, y), as shown in Eq.(8):

Where c₁ is a constant (here c₁ = 1) to avoid singularity. Based on the theory of references [22, 23], every S _x and S _y can respectively be calculated by using formula: \( {S}_x=\sum \limits_{i=1}^W{x}_i \) and \( {S}_y=\sum \limits_{i=1}^H{y}_i \) where x _i is the intensity of every row and y _i is the intensity of every column. W and H is the width and height of every part.

Contrast: Contrast function can be designed as c(x, y). It is similar to Eq.(8), but it uses σ _x and σ _y to express the estimation of the contrast, as show in Eq.(9):

Where c₂ is also constant (here c₂ = 1) to avoid singularity.

If the intensity average value of every row is S _i , the standard deviation σ _x can be calculated by using formula: \( {\sigma}_x=\sqrt{\frac{1}{N-1}\sum \limits_{i=1}^N\left({x}_i-{S}_i\right)} \) and σ _y can be gotten with the same form.

Structure: Structural function can be designed as s(x, y). It uses the correlation between the ideal pattern and dithering pattern to measure the structural similarity, as shown in Eq.(10):

Where the c₃ is similar to c₁ and c₂ (here c₃ = 1) and the standard deviation σ _xy is solved for \( {\sigma}_{xy}=\frac{1}{N-1}\sum \limits_{i=1}^N\left({x}_i-{S}_x\right)\left({y}_i-{S}_y\right) \).

The local weighted function (LWF) is generated by weighted combing the luminance function、contrast function and structural function, as shown in Eq. (11)

Where α, β and γ are used to be weighted between luminance function、contrast function and structural function. Usually α, β and γ are set to 1.

The ideal sinusoidal fringe pattern I(x, y) and dithering pattern I _d (x, y) have the same process to calculate the LSF and LSF _d , and finally the function SSIM(I, I _d ) can be calculated as shown in Eq.(12):

Where I(x, y) is the pixel intensity of desired ideal sinusoidal fringe. I _d (x, y) is the pixel intensity of the Gaussian filtered dithering fringe.

Phase-based optimization (p-opt) and intensity-based optimization (i-opt) are also two main categories to improve the quality of fringe patterns. In order to show the superiority of the proposed optimization method, we compare it with the norm Frobenius (NF) [17], the p-opt [18], i-opt [19] and ideal sinusoidal fringe patterns under the same condition. Figure 1 shows the effectiveness of the proposed method. Figure 1a shows the cross-section comparison between the ideal sinusoidal curve and fringe patterns with different optimization. Figure 1b and c give the intensity difference and phase difference with p-opt, i-opt, NF and the proposed method. From the results we can find that the fringe pattern based on phase optimization has less intensity difference than that based on intensity optimization. It means phase-based optimization can generate better quality fringe pattern than intensity-based optimization. Compared with these two methods, the proposed method has the least intensity difference which means the quality is the best among these optimization methods, while the quality based on the norm Frobenius is the worst because intensity difference is the largest. By analyzing the phase errors, it can also be found that the phase error based on NF is the largest. Phase-based optimization is superior to the intensity-based optimization while the proposed optimization can generate the least phase errors. It proves the advantage of dithering fringe pattern after optimization of WSF. It can generate better quality fringe pattern with less intensity difference and less phase errors.

The cross-section fringe comparison of different optimization. a: The comparison between ideal sinusoidal curve and cross section fringe pattern with different optimization. b and c: The intensity difference among four methods. c: The phase error comparison among four methods

Full size image

Because NF just compares the global similarity without considering the local structure similarity and other methods are designed for the local details, in the following section we just compare the intensity-based optimization, phase-based optimization and the proposed optimization method.

Implementation in the whole fringe optimization framework

Figure 2 is a flowchart showing the proposed algorithm implemented in the whole fringe optimization and the optimization process is implemented as:

A flowchart showing the proposed algorithm implemented in the whole fringe optimization

Full size image

Step 1: Intensity error pixel detection. Intensity absolute error map ΔI(x, y) is calculated by ΔI(x, y) = |I(x, y) − I _d (x, y)|, where I(x, y) is the intensity of ideal sinusoidal fringe and I _d (x, y) is the intensity of Gaussian filtered binary fringe. When the pixels with intensity error are bigger than a certain threshold, they can be defined as error pixels. In this paper, ten fringe patterns are projected to a flat and a series of images are captured. The same period of sinusoidal fringe pattern is designed and compared with captured image. The mean difference value is used as the threshold.

Step 2: Error pixel reassignment. Error pixels are mutated from 0 to1 or from 1 to 0. After mutation of each error pixel, the binary fringe is filtered and global intensity RMS error is calculated. Only good mutations are kept, and bad mutations are discarded. Here good mutation means after this mutation, the global intensity RMS error is reduced.

Step 3: Iteration. As Gaussian filter is applied to the fringe patterns, the error pixel mutation of each pixel will influence its surrounding pixels. So after error pixel reassignment optimization of the whole fringe, steps 1 and 2 are carried out iteratively. Here the iteration process will continue until the intensity RMS error improvement is less than 0.1% in one round or the iteration reaches maximum iterating times.

Step 4: Local error pixel detection. LSF is applied on the ideal sinusoidal fringe patterns to obtain the LSF of ideal sinusoidal fringe patterns. LSF is also applied on the filtered sinusoidal fringe patterns to obtain the LSF _d of dithering fringe patterns. The absolute error between LSF and LSF _d is calculated by using SSIM(I, I _d ). When absolute error of the pixels are larger than the threshold, they are defined as error pixels.

Step 5: Local error pixel reassignment based on the WSF. Pixels in three shifted fringe patterns are optimized simultaneously. If local error pixel is marked as an error pixel, the corresponding pixels in the three dithering fringes will mutate their binary status from 0 to 1 or from 1 to 0. There are eight kinds of combination:(0,0,0), (0,01), (0,1,0), (0,1,1), (1,0,0), (1,0,1), (1,1,0), (1,1,1). The WSF error is calculated for every combination and the smallest error combination is kept. This algorithm is applied to each local error pixel.

Step 6: Iteration. After local structure optimization for the whole fringe patterns, steps 4 and 5 are carried iteratively, until the intensity RMS error improvement is less than 0.1% in one round or the iteration reaches maximum iterating times.

The effectiveness of the proposed method is verified through simulation. In practice, the measurement quality is ultimately determined by phase quality of the binary dithering fringes. The simulation is performed for fringe patterns with different defocusing levels to test the phase RMS error. Because (n + 2)-step phase-shifting is insensitive to the n order harmonics, ten-step phase-shifting technology is used as the reference. The sinusoidal pattern with the same period is used as the reference data to calculate the phase RMS error. The phase RMS error is calculated after applying different sizes of Gaussian filter. In this paper, the Gaussian filter size of 5 × 5 is used to represent slightly defocused and 13 × 13 represents strongly defocused. We use Gaussian filter of sizes G = 5 ∼ 17 and σ = G/3 to simulate the defocused projector. In this simulation, the phase is obtained by using a three-step phase-shifting algorithm with equal phase shifts. The phase RMS error is calculated by taking the difference between the phase obtained from the smoothed binary patterns and the phase obtained from the ideal sinusoidal fringe patterns. Figure 3 shows the phase RMS error comparison among the phase optimized fringes [18], intensity optimized fringes [19] and the proposed fringes of whole-fringe optimization. The period T ranges from 20 pixels to 90 pixels (T = 20,30...90). When the period is small the intensity-based optimization has less phase RMS error than phase-based optimization, and the proposed optimization has the same tendency with intensity-based optimization but it is superior to the intensity-based optimization with different Gaussian filter sizes. When the period is increased, the phase RMS error has little change. It shows the optimized dithering fringe pattern can be used not only for small period but also for large period fringe pattern. Especially when the period is very large such as T = 90 pixels, the proposed optimization has the least phase RMS error compared with the other two methods. It shows that the proposed method inherits the features of dithering fringe pattern and it has the advantage to be used for large range 3D shape measurement with wide fringe pattern.

The phase RMS error comparison among the phase optimized fringes, intensity optimized fringes and the proposed fringes of whole-fringe optimization

Full size image

By analyzing of the simulation, we can also find that when the projector is slightly defocused (Gaussian filter size is 5 × 5), the phase-based optimization has the lowest phase RMS error while with the Gaussian filter size increasing, the quality of phase optimized fringes decreases due to the sensitivity to the filter size. The phase error of the intensity-based optimization fringes and the proposed fringes decrease sharply as filter size increases, and they outperform the phase optimized fringes. Because the proposed method focuses on the local structure similarity, the proposed fringes have the same tendency with the intensity optimization fringes and it outperforms the intensity-based optimization fringes on phase quality with all filter sizes.

We also develop a 3D shape measurement system to verify the proposed method. As shown in Fig. 4, in this system, we utilize a DLP projector (Samsung SP-P310MEMX) and a digital CCD camera (Daheng MER-500-14U3M/C-L). The camera is attached with a 16 mm focal length lens (Computar M1614-MP) with an image resolution 1024 × 768. The projector has 800 × 600 resolution and 0.49–2.80 m projection distance.

3D shape measurement system

Full size image

The first experiment is to reconstruct a flat white board with the intensity-based optimization fringes pattern, the phase-based optimization fringe pattern and the the patterns base on our method. The projector and the camera are operated under exactly the same condition during all experiments. The captured fringe patterns by the proposed method are shown in Fig. 5a and the projector is set to slightly defocused. Figure 5b shows the reconstruction result of the plane. And the phase errors of the reconstruction results with different methods are shown in Fig. 5c. Again, the proposed method generated the best results while the intensity-based optimization method performs worst.

The flat white board is measured and results comparison based on the intensity-opt, phase-opt and proposed-opt

Full size image

To verify the the robustness of the proposed fringe pattern under different defocusing levels and evaluate the measurement quality, a more complex 3D mask is measured. In practice, it is hard to give quantitative levels, so in this paper, the projector defocusing levels are set: nearly focused, slightly defocused and strongly defocused. Because dithering fringe pattern has the advantage for large-range 3D shape measurement with wide stripes, in this measurement the period T is set to 90 pixels. The measurement results are shown in Fig. 6.

The measurement comparison among three methods under nearly focused, slightly defocused and strongly defocused

Full size image

From the measurement results, we can find that when the projector is nearly focused, measured results of all the optimized binary fringes are strongly corrupted by noises but the phase-based optimization and the proposed optimization fringe pattern can get better results with less noise. In order to compare the measurement result RMS error, we use the result from 9-step shifted fringe patterns as reference. The measurement results comparison under different projector defocusing levels are presented in Tables 1, 2 and 3. When the projector defocsing level increases, the phase error of all the optimization decreases in general and the phase error of intensity-based fringe and the proposed fringe are larger than phase-based optimization. The RMS of phase-opt, intensity-opt and the proposed-opt are 0.75 mm, 0.62 mm and 0.43 mm, respectively. With the defocusing levels increasing, when the projector is slightly defocused and strongly defocused, the dithering binary pattern will get smoother and the proposed method will get more accurate measurement results than the phase-based and intensity-based optimization methods. Under slightly defocused, RMS of phase-opt, intensity-opt and the proposed-opt are 0.62 mm, 0.55 mm and 0.42 mm, respectively and under strongly defocused, they are 0.42 mm, 0.42 mm and 0.38 mm. From the measurement results and error comparison, we can find that the phase-based method and proposed method perform better than the intensity-based method under the nearly focused for optimization, while the proposed method and intensity-based method steadily improves the accuracy with increasing amount of defocusing. Especially under the slightly defocused and strongly defocused, they are superior to the phase-based method. By comparison and analysis, because we combine the global similarity and local similarity, the proposed optimization fringe pattern has better measurement quality and less RMS error from nearly focused to strongly defocused. This experiment also conforms with our simulation results. It demonstrates that the proposed method can achieve high quality fringe patterns when the projector is at different amounts of defocusing and the proposed method outperforms the phase-based method and the intensity-based method.

Recently the optimization methods just qualify the global similarity of the pattern while the global optimization methods ignore the influences of local structure. In order to get high-quality binary dithered patterns, we propose a dithering optimization techniques by combining global similarity and local structure similarity. Both simulation and experimental results have demonstrate that the proposed algorithm can achieve binary fringe pattern with high measurement quality and it is robust to the amounts of defocusing.

3D:

Three-dimensional

i-opt:

Intensity-based optimization

p-opt:

Phase-based optimization

This work was supported by the Fundamental Research Funds for the Central Universities (N150403009) and the Fundamental Research Funds for the Central Universities (N162610004).

Funding

Faculty of Robot and Engineering of Northeastern University, Shenyang, China provides the funding for this research.

Availability of data and materials

Details data has been provided in this paper.

Affiliations

1. Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China

   • Feng Lu
   •  & Chengdong Wu

2. Ophthalmology, General Hospital of Shenyang Military Region, Shenyang, China

   • Jikun Yang

Contributions

All authors contribute equally in all the sections of this work. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Feng Lu.

Authors’ information

Feng Lu, a doctor of Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China. His interest includes Structured light, Robot and image processing.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Reprints and Permissions