Improving the calibration of phase measuring deflectometry by a polynomial representation of the display shape

Abstract

Background

Phase measuring deflectometry is a highly precise and full field metrology technique for specular surfaces based on the distortion of known reference patterns observed as a reflection at the surface under test. Typically, liquid crystal displays are employed to provide the required patterns. Due to a lack of research, these displays are used without sufficient calibration.

Methods

In this work, we present an enhanced calibration for phase measuring deflectometry, taking flatness deviations of the display surface into account. The display shape is modelled as a polynomial surface whose coefficients are determined by minimizing the retrace error in a global optimization procedure during calibration. This approach does not require any additional measurements or hardware. Improvements due to the enhanced calibration model are qualified experimentally using a flat and a spherical concave mirror.

Results and conclusion

The model-based parameterization of the display surface yields significant improvement on both samples. The peak to valley (PV) of measured deviations on the plane mirror are reduced by 67% to 0.55 μm. Measuring the spherical sample without the display parameterization leads to a rather large shape deviation of 33.40 μm PV which is reduced by 94% to 1.98 μm. The viability of our approach confirms the dominant role of flatness deviations of the display surface as an error source in absolute shape measurement using phase measuring deflectometry.

Introduction

The precise, contact-free and full-field measurement of reflecting surfaces still poses a challenge to modern metrology which is in most cases answered by employing interferometric techniques. However, coherence-based techniques usually suffer from some fundamental difficulties, for example a high sensitivity to vibration, rendering their application within an industrial environment a difficult task. Furthermore, the need for sophisticated components and specific optics leads to high costs. Deflectometry, in contrast, provides a robust and low-cost alternative. For a comparison of both techniques see e.g. [1,2,3,4]. Deflectometry measures the shape of specular surfaces by tracing the reflection of a structured, incoherent light source. It can be applied as a full-field measurement by using a camera as sensor, capturing reflected images from an extended, known reference pattern. Phase Measuring Deflectometry (PMD) employs multiple series of phase-shifted sinodial-patterns that provide a unique coding of the surface used to display them and allows to measure the geometry of the surface under test [5]. Typically, commercially available liquid crystal displays (LCDs) with their high manufacturing quality [6], commonly employed as computer screens, are used as structured light source to provide the required patterns due to their ability to display user-defined patterns with good contrast at a sufficient frame rate. A detailed overview regarding the state of the art is given by Huang et al. [7].

In our preliminary investigations [8], we studied the errors introduced by non-ideal display properties assessing the shape of flat surfaces in a simulative approach. We modelled angle dependent emission characteristics, refraction at the cover glass, pixilation, limited color depth and shape deviations. We clearly identified flatness deviation of the display as the most significant error source in PMD that introduces low spatial frequency deviations when assessing the global shape of the surface under test.

To account for this error source, we enhanced our calibration and evaluation procedures by introducing a polynomial model of the display shape. The corresponding coefficients are determined during the calibration procedure by minimizing the retrace error. Hence, this approach does not require any prior knowledge regarding the shape of the display surface. We present a quantitative investigation of this approach using an optical flat and a spherical concave mirror as surfaces under test. For both samples, we achieve a decreased shape deviation with respect to its specifications.

Methods

Phase measuring Deflectometry (PMD)

Figure 1a shows the basic principle of deflectometry. Patterns from a reference surface are observed by a camera as a reflection at the surface under test. Due to the shape of the specular surface, the recorded patterns are distorted, enabling assessment of shape information. Phase Measuring Deflectometry (PMD) employs series of phase-shifted sine-patterns, typically using a computer display as reference surface. This approach provides the observed position on the display for each individual camera pixel by applying the well-known phase shifting technique to the recorded images of fringe patterns seen as a reflection in the surface under test [9]. The process of capturing these fringe images and calculating the positions on the display is therefore referred to as phase measurement. System calibration provides the relative positions and orientations of the camera and display. Also, each camera pixel is assigned to a so-called vision ray, giving a geometric description of the inverse path of received light. This information is either given by a set of parameters describing all vision rays based on a photogrammetric approach or, as it is the case here, by a generic description providing independent vision ray information for each camera pixel [10].

Fig. 1

a) Principle of Phase Measuring Deflectometry: A display shows a pattern which is observed by a camera as reflection at the surface under test, distorting the image. Displaying multiple phase-shifted sine pattern enables phase measurement to trace each camera ray to the display. b) Calibration procedure: A confocal white-light distance sensor marks a reference point in space which is on the surface of a plane calibration mirror. Phase measurements for multiple rotated orientations of the mirror around arbitrary axis through the reference point enable determination of position and orientation of camera, display and reference point

The geometry of the surface under test is assessed by identifying surface positions and its local slopes leading to a valid path of light between each camera pixel, the corresponding position on the specular surface and the observed position on the display. However, there is an infinite amount of combinations of surface positions and its corresponding local slopes yielding such a valid path [11]. The process of identifying the correct combinations is referred to as regularization and requires additional information. There exists a variety of techniques for the assessment and utilization of such information to solve this ambiguity [12, 13]. In this work, we approached regularization by knowledge of a reference point on the surface, serving as a seed point for the reconstruction [14, 15]. A distance sensor marks this reference point in space and the calibration procedure described below provides its absolute coordinates.

Figure 1b sketches the calibration procedure: A plane calibration mirror replaces the surface under test. Phase measurements are performed for multiple tilted orientations of the mirror, allowing to apply bundle adjustment to determine the relative positions and orientations of camera, display and all mirror poses used [16]. By additionally keeping the reading of the distance sensor constant while aligning the mirror, we assure each mirror reorientation to be a rotation around an axis through a common point in space. Adding the knowledge about the existence of this unknown common point as a constraint to the bundle adjustment allows to additionally determine the coordinates of this reference point during calibration. A calibration approach similar to ours, however, using a different approach for regularization is described in detail by Xiao et al. [17]. To measure an unknown sample with the calibrated system, the surface under test is placed considering the distance reading used during calibration, ensuring the systems reference point being an actual point on the surface. Initial values for surface locations are acquired by calculating the intersections of the vision rays from the camera with a plane through the reference point. Typically, one of the determined poses of the plane mirror used during calibration is utilized to provide this reference plane. Additionally, phase measurement provides knowledge about intersection points of camera vision rays with the display. This enables the calculation of gradients at each point of the currently assumed surface intersected by a vision ray. Unless assumed surface points are actually true, the law of reflection, postulating equal angles of incidence and reflection, will not be satisfied. However, integration over the assessed gradients yields a more accurate surface description than the initial guess. This enables an iterative procedure of computing intersection points of vision rays with the currently assumed surface shape, calculating the corresponding gradients and integrating these to determine the true shape of the surface [18]. The reference point, known to be an actual point on the true surface and immediately providing valid gradient values, provides the starting point of each integration.

Distance measurements are performed using a confocal white light sensor, enabling an accuracy of a few micrometers on specular surfaces tilted up to 5°. All other techniques known to the authors fail at this task since they either rely on normal incidence or diffuse scattering. Alternatively, since no absolute distance readings are required, a reference point can be marked in space by crossing two laser pointers [14]. This approach however cannot keep up with the accuracy of a confocal sensor and will introduce systematic, low spatial frequency shape deviations to the measurement.

The calibration procedure outlined above does not require preliminary knowledge regarding the geometry of the setup beyond the mirror used for calibration being flat and all its poses sharing one common point in space. Relative positions and orientations of camera, display, reference point and mirror poses are determined solely based on recorded phase measurements. This allows to easily set up or adjust and calibrate a measurement system without the need for additional metrology technique. The drawback of this approach is at the same time the reason for our investigations: The geometric calibration is already influenced by error sources affecting the deflectometric measurement and compensating these errors by adequate models of the components of the setup is of particular importance. Alternatively, a deflectometry setup can be calibrated using external metrology systems like laser tracker and coordinate measurement machines as demonstrated by Huang et al. [19]. Such a calibration typically exhibits higher accuracy compared to our approach at the cost of highly increased effort and hardware requirements.

Display model

For a mathematical description of the display, we define a native coordinate system for its surface. Assuming a flat display and using its center as origin, it can be parameterized as

$$ z\left(x,y\right)\kern0.5em =0\kern0.5em \forall \kern0.5em x\kern0.5em \in \kern0.5em \left[-\frac{W}{2},+\frac{W}{2}\right],y\kern0.5em \in \kern0.5em \left[-\frac{H}{2},+\frac{H}{2}\right] $$

(1)

with W and H being the width and height of the display respectively. Coordinates in this native system are related to those of equivalent native systems of camera and reference plane by homogeneous transformation matrices determined during calibration. Therefore, absolute coordinates of intersection points of vision rays with the display can be computed from measured and unwrapped phase angles, providing (x, y)-coordinates in the display system due to their known relation to the employed wavelengths of the fringe patterns.

To leave the approximation of a flat display surface while assuming the shape to only vary slowly, Eq. (1) is modified by a bivariate polynomial, yielding

$$ z\left(x,y\right)=\sum \limits_{i>1}^N\sum \limits_{j>1}^N{c}_{ij}\cdot {x}^i{j}^j\kern0.5em \forall \kern0.5em x\kern0.5em \in \kern0.5em \left[-\frac{W}{2},+\frac{W}{2}\right],y\kern0.5em \in \kern0.5em \left[-\frac{H}{2},+\frac{H}{2}\right] $$

(2)

Here, N is the chosen maximum order of the polynomial, c_ij are the coefficients. The later are initially unknown. Zeroth and first order terms of the polynomial are ignored on purpose since these represent offset and tilt of the display which is already assessed while determining the homogeneous transformation matrix relating the display system to other system components. In our approach, the calibration is performed under the assumption of a flat display in the first step. Using these results as initial values we rerun the calculation using the display parameterization from Eq. (2) with the coefficients c_ij added as additional variables to the bundle adjustment. We are well aware that the determination of display coefficients by minimizing the retrace error will be affected by other non-ideal display properties, e.g. refraction at the cover glass, which are not yet considered in the model. Despite these shortcomings and due to the dominant role of the shape deviation in comparison to other non-ideal display properties [8], our extended calibration model can still yield significant improvements regarding the accuracy of global shape measurement.

Results

The setup used in our investigations employs a camera of type AVT Pike F210B with a resolution 1920 × 1080 pixels and an objective of type Kowa LM35SC with a focal length of 35 mm. Our display, the NEC MD211G5, is a grayscale-monitor designed for applications in medical diagnostics, providing better contrast and less angle-dependencies of emissions compared to common consumer displays. Distance measurements used to mark our reference point in space are performed using a confocal white light sensor of type IFS2405–28 with IFC2461 controller, both manufactured by micro-epsilon. An optical flat with certified λ/20-flatness is used as calibration mirror. Twenty-five phase measurements for varying mirror orientations are recorded to calibrate the system. The calibration is calculated using the flat display model given by Eq. (1) as well as with the polynomial model in Eq. (2) with a 7th order polynomial. Because display shape and location and orientation of the components of the setup are determined in a global optimization procedure, we yield overall different results for the setup geometry by calibrating with both model assumptions. For validation, both calibration results are applied to the phase measurement of one exemplary mirror orientation to compare yielded shape deviations of the optical flat. Figure 2 shows the results. Assuming an ideally flat display surface, see Fig. 2a, leads to shape deviations of 1.65 μm peak to valley (PV) and 210 nm root mean square (RMS). Application of the calibration result assuming a polynomial display surface reduced the measured shape deviation to 0.55 μm PV and 124 nm RMS as can be seen in Fig. 2b.

Fig. 2

Measured shape deviations of the plane calibration mirror based on a) calibration assuming a plane display surface and b) using a polynomial display model of 7th order whose coefficients are determined during calibration. Both calibrations and shape evaluations were performed on the same phase measurement data. Measured shape deviation is reduced from a) PV = 1.65 μm, RMS = 210 nm to b) PV = 0.55 μm, RMS = 124 nm by application of the polynomial display model

The reduction of measured shape deviations on the flat mirror on its own is not sufficient for the validation of improvements due to the polynomial display model because it is the working principle of the calibration to identify those system parameters which yield minimum retrace error under the assumption of a flat surface as specular object. Therefore, the resulting display shape is not necessarily related to the actual display but could instead just be caused by compensating the effects of various other error sources. To validate the improvement due to the polynomial display model, the experimental setup is additionally used for shape measurement of a spherical concave mirror with radius R = 203.2 mm while utilizing both calibration results for evaluation of measured phase data. The shape deviation of this sample is determined with respect to the specified surface shape. The results are displayed in Fig. 3: Assuming a flat display surfaces yields the shape deviation shown in Fig. 3a with 33.40 μm PV and 8.23 μm RMS. Using the system calibration taking display flatness deviations into account, shape deviation is reduced to 1.98 μm PV and 0.46 μm RMS. as shown in in Fig. 3b. The reduction of measured shape deviation on this sample, which was not used for the determination of geometry parameters of the setup, states that the calibration using the polynomial display model in fact yields a more accurate result.

Fig. 3

Measured shape deviations of spherical concave mirror with radius R = 203.2 mm based on a) calibration assuming a plane display surface and b) using a polynomial display model of 7th order whose coefficients are determined during calibration. Both calibrations and shape evaluations were performed on the same phase measurement data. Measured shape deviation is reduced from PV = 33.40 μm, RMS = 8.23 μm in a) to PV = 1.98 μm, RMS = 0.46 μm in b) by application of the polynomial display model

Further validation of our calibration approach is achieved by evaluation of the actual display shape determined in the calibration. This evaluation must be qualitatively in this work, because the surface shape of a display is influenced by temperature as well as gravity and physical stress. However, we have to assume constant display shape in calibration and all subsequent measurements and results can only be approximations. Figure 4 shows the shape of the display surface determined by identifying the coefficients for the polynomial parameterization in Eq. (2) yielding minimum retrace error. Each of the images of Fig. 4a to c represents the full surface of the display which has a resolution of 2560 × 2048 pixel with pixelpitch 165 μm. Black areas correspond to display surface regions which were not observed during phase measurement. Color-coding marks the deviation in normal direction regarding an ideal plane. Figure 4a shows the entirety of the display shape information yielded by the calibration. Phase values near the frame of the display are masked out in all measurements since these are prone to errors. This causes the display shape to be undetermined in these regions. Additionally, some areas near the corners were not observed by any of the 25 orientations of the plane calibration mirror used to determine the calibration. The calibrated display surface exhibits a PV of 0.94 mm. Figure 4b shows the observed display surface area and corresponding shape information for the one exemplary orientation of the plane calibration mirror used in Figure 2. Display shape data is taken from Fig. 4a, masked in accordance with the observed display area and tilt- and offset-corrected by subtraction of the best-fit-plane. This yields a remaining shape deviation of PV = 0.32 mm. The white frame outlines the calibrated display surface area from all calibration measurements in Fig. 4a. Analogous to the representation in Fig. 4b, c shows the observed display surface area and shape for the measurement of the spherical concave surface in Fig. 3. The field of view of the camera is focused to a relatively small area on the display due to the reflection at the concave mirror. This display surface area exhibits a shape deviation of only 0.05 mm PV.

Fig. 4

Determined shape deviation of the display surface. Images a) to c) represent the full display surface (resolution: 2560 x 2048 pixels). Deviations in normal direction to the best-fit-plane are color-coded. Black areas were not accessed by the phase measurement. a Display shape from system calibration based on 25 phase measurements with the plane calibration mirror. Measured phase values near the display frame were masked out since these are prone to errors. Some regions near the corners were not accessed by phase measurements. PV of display shape deviation over the calibrated area is 0.94 mm. The outline of the calibrated display surface is drawn as white frame in b) and c). b Display shape deviation for the observed surface area from an exemplary phase measurement of the plane calibration mirror, derived from determined display shape in a). PV of display shape deviations over the observed area is 0.32 mm. c) Display shape deviation for the observed surface area from phase measurement with the spherical concave sample as surface under test, derived from determined display shape in a). PV of display shape deviations over the observed area is 0.05 mm. The observed display surface areas in b) and c) correspond to the shape measurements of the surface under test in Figs. 2 and 3 respectively

As a reference to the display shape determined by our calibration, we measured the shape of the display used in our experiments using a coordinate measurement machine of type Mitutoyo Crysta-Apex C. The results are shown in Fig. 5. The PV of measured shape of 0.54 mm is smaller than the corresponding result from our calibration in Fig. 4 but exhibits the same order of magnitude. Distinct accordance of both results for the surface shape cannot be found. On one hand, we can assume this to be caused by in fact diverging surface shape of the display: During the tactile measurement, the display was turned off and lying on the table with its surface normal pointing upwards. In contrast, the display was in upright position during our experiments and also operating and thus generating heat. On the other hand, determination of the display shape during calibration is based on model assumptions that neglect further effects such as that of the cover glass or the area of confusion. Even though we must assume that the shape of the display is not determined accurately during calibration, we find that we yield improved measurement accuracy by calibration using the polynomial display model, assumedly caused by an overall improved calibration result.

Fig. 5

Shape of the display surface determined with a coordinate measurement machine. The PV of 0.54 mm is smaller than those of the shape determined during calibration (see Fig. 4) but is in the same order of magnitude. Distinct accordance of both results is not found. However, the display shape is influenced by temperature, gravity and physical stress. While using the coordinate measurement machine, the display was turned off and positioned facing upwards. During our deflectometric measurements, the display was running and placed upright

Discussion

Flatness deviations found for our display by identifying model coefficients in a maximum likelihood approach (see Fig. 4a) are not in strict accordance with those determined using a coordinate measurement machine (see Fig. 5) but are in the same order of magnitude. Since the shape of the display is in fact influenced by temperature and orientation, precise accordance cannot be expected.

Shape measurements using the enhanced system calibration yield significantly decreased deviations compared to those calculated based on the assumption of an ideally flat display surface as can be seen in Figs. 2 and 3. The PV of measured shape deviation on the plane calibration mirror is reduced by 67% to 0.55 μm. While the magnitude of deviations is decreased, their general form remains mostly similar. In case of the spherical concave surface, the PV of measured deviations with respect to the specified shape is reduced by 94% to 1.98 μm. While the form of the deviation resembles a paraboloid when being evaluated with the default calibration, application of the calibration containing the display parameterization yields deviation that can be described as freeform with some symmetries with respect to an axis through the center. Due to the concave shape of the object, the field of view of the camera is focused to a relatively small area on the display which only exhibits a flatness deviation of about 50 μm. However, we can assume the localization of this surface area within the global coordinate system of the setup to be more accurate when a non-flat display is considered during calibration.

Comparing the best results of both surfaces under test, shown in Figs. 2b and 3b, we find that the deviations measured on the spherical surface are about 4 times higher than those measured on the plane calibration mirror. This seems plausible because curved surfaces are generally harder to measure using PMD since the circle of confusion of individual camera pixels on the display becomes distorted on reflection at the curved surface. On the other hand, we find the relative improvement of measured shape deviations due to the display calibration to be significantly larger for the spherical surface. Currently, we cannot explain this phenomenon and are planning to address this issue in future research.

The accuracy of the display shape calculated by minimizing the retrace error depends on quality and completeness of the models involved. E.g., we currently ignore the effects of refraction at the cover glass which also contributes to the retrace error affecting our calibration. The significant improvements due to the display calibration presented here confirm the dominant role of the display shape on systematic measurement errors in PMD found in our preliminary simulations [8].

Finally, assuming constant shape of the display surface over time implies another approximation because of thermally driven changes during operation. To limit this effect, all measurements presented here, including those used for the system calibration, were recorded within one session after a warmup-phase of about 2 h. We still expect a systematic error due to this effect which we cannot quantify yet. Compensating the effects of a non-constant display shape would require an alternative approach were the display shape is perpetually monitored. This could be realized by introducing an additional photogrammetric measurement system directly observing the display and independently recording the series of fringe images, providing display shape data for every measurement of the PMD setup.

Conclusions

Assuming a plane display surface in PMD causes severe errors on absolute shape measurement. Introducing a simple polynomial model whose coefficients are determined by minimizing the retrace error during calibration yields reasonable results for the display shape and significantly reduced systematic measurement errors. The viability of this approach confirms the finding of our preliminary simulations [8] which identified flatness deviations of the display as the major error source in PMD.