Energy balance in Spectral Filter Array camera design

Acquisition model and balanced sensor

This equation could also include the transmission of the medium in which the system is used, e.g. underwater imaging.

We define energy balance of a sensor for one illuminant when, for a perfect diffuser, the variance of Π is equal to 0, in other words, when ρ ₁=ρ ₂=…=ρ _N . The flat reflectance object may be replaced by a specific reflectance property characterizing the application. For compactness, we denote Π=X(F), a function where X define a vector Π from a set of filters F, according to the imaging model of Eq. 1.

The solution where this converges might not be suitable for multispectral acquisition. Indeed, an obvious solution to this problem is when F is composed of the same filters (i.e. f ₁=f ₂=…f _N ). To ensure the practicality of the filter set, we constrain the set F to a reduced subset of T. The reduced set is comprised of Gaussian filters distributed over the considered part of the spectrum. Then, F is characterized by its set of peak sensitivity locations Ω=μ ₁,…,μ _N , which is fixed by the application. We propose to give a freedom to two parameters in order to find an optimal F using the standard deviation of each Gaussian σ ₁,…,σ _N and the maximum transmittance B ₁,…,B _N . This model could be adapted to several models of transmittance, but for clarity of communication, it is easier to work with Gaussian filters.

According to Definition 1, one may still find a minimum convergent solution. However, the set might not be viable in practice, due to limited efficiency or unpractical width. According to Definition 2, we can not converge to zero due to the variability in illuminations. But we can give a degree of freedom, ε, such as Y ^′(F)+ε=0.

In this context, we must put constraints on the model, in order to ensure the practicability of the filters set (and its realization), a reasonable variation among the set and relatively good maximum transmittance. These set of constraints are to be defined in next Sections.

Sensor considerations

A major element in this system is the shape of the sensor spectral response S(λ). The silicon response is a fixed parameter in our methodology because we assume the choice of sensor to be made a priori, based on the application. An example of such a characteristic is presented in Fig. 2.

A photon is characterized by a wavelength λ which is equivalent to an energy E. The inverse relationship between the energy of a photon and the wavelength is defined by:

$$ E=\frac{hc}{\lambda}, $$

(3)

where c is the speed of the light in vacuum and h the Planck’s constant.

Photons with an energy bellow 1.1e V will go through silicon without interaction [35]. This corresponds, according to Eq. 3, to a theoretical wavelength larger than 1125 n m. The structure of the sensor will, in fact, reduce this wavelength prior to reaching the theoretical limit [35]. Our model does not include the optical effects. Indeed, the layer of silicon can act as a Fabry-Pérot interferometer [36], which modifies the absorption properties of the substrate. Without the optical effects, we can use directly the spectral response as defined in Fig. 2.

Filter considerations

The filters considered in SFAs are most often a realization of a Fabry-Pérot interferometers today [5, 7, 37]. Future technical development may give rise to nano-tube or nano-hole SFA realizations [10]. Filters can be metallic filters [38, 39] or nano structure filters [40], and the substrate is often silicon or doped silicones, germanium, AsGa, indium, etc.

We use a simple Gaussian model to approximate Fabry-Pérot interferometric filters, such as realized by SILIOS [37] and IMEC [8]. This approximation has been extensively done in the literature, in Fig. 3, we verify the impact of this approximation in term of energy inaccuracy before to push further our description. This figure shows the difference between a Gaussian approximation (G), a model of Fabry-Pérot (FPM), and measurement of a practical Fabry-Pérot designed by SILIOS (FPR). If we compute the differences between these curves as integral of function difference, we obtain the following numbers: d(F P R,G)=36.74%, d(F P M,F P R)=9.96% et d(F P M,G)=46.70%. Although the difference between G and the Fabry-Pérot instances seems to indicate that the approximation may not be adequate, the difference between the FPM and the FPR indicates that depending on the technology, general models would not be adequate either due to noise and specific -secret- tuning of the manufacturer. We then make the statement that for illustration purpose it is more interesting to develop a methodology with Gaussian and to let the user of this methodology to tune it toward his own model of filters. In addition, if we only compute the difference in peak values, the Gaussian model is closer to the Fabry-Pérot model than the measurement, due to the realization process. If we consider the differences between these curves within the standard deviation of the Gaussian, i.e. full width at half maximum (FWHM), the difference between FPM and FPG is greater than the difference with G and validate the use of the model by the assumption that the difference comes from the lobes of the Fabry-Pérot filter, which generate only an offset spectrally consistent. This simple computation may also serve in the analysis of literature results obtained with Gaussian simulations.

The shape of the filter according to the Gaussian model depends on three parameters, namely the standard deviation σ, the scaling factor (the peak efficiency) B, and the average of the distribution (the location of the filter in the electromagnetic range) μ. Example of filters are given in Fig. 4 where we show a set of three Gaussian filters uniformly distributed. Within this model, we can write f _i as in Eq. 4.

$$ \begin{array}{l} f_{i}(\lambda)=B_{i}\frac{1}{\sigma_{i} \sqrt{2\pi }}e^{-\frac{1}{2}\left(\frac{\lambda-\mu_{i}}{\sigma_{i}}\right)^{2}}\\ \end{array} $$

(4)

where μ _i are fixed parameters according to the application and σ _i and B _i correspond to the given degree of freedom. The distribution of μ may be given a degree of freedom, but according to our knowledge, users usually optimize it for a specific application. Here, we consider that μ are provided in a priori design and as example, we consider that it is an equi-distributed set or a list of selected peaks, such as shown in the Results section. In some cases, where μ are not provided or in a more global framework, it may be embedded into the optimization process with little extra work and restrained so that all channels do not end with the same transmittance.

As for example purpose, we define an equi-distributed μ set, within the global applicable range of wavelengths Δ Λ=λ _max −λ _min and the number of channels N, so that the central positions μ _i of each Gaussian are fixed before the optimization process. Consequently, in this case, the position μ _i can be defined like in Eq. 5:

$$ \begin{array}{l} \mu_{i}=\lambda_{min}+\frac{\Delta \Lambda(2i-1) }{2N},\\ \end{array} $$

(5)

In a practical realization of an SFA system, we may consider two situations, as shown in Figs. 5(a) and (b).

We can consider two sensor configurations. One is very specific to some applications, such as in medical imaging [41, 42], where a very narrow spectral band of about 10 nm or ratio computed between specific narrow bands are required. In this case, Fig. 5(b) is a typical setup, and peaks cannot be given freedom. Another situation, where very wide bands are considered, does not to provide an optimal results either for demosaicing or for spectral reconstruction. After looking at the results of Sadeghipoor et al. [33] and Wang et al. [43], it seems that rather a relatively wide band is adequate for general demosaicing and spectral reconstruction. Indeed, we want to preserve separability between wavelengths if possible, e.g. NIR channels must not be polluted by visible information for computer vision, while maximizing spectral correlation for demosaicing. In this last case, we may authorize a little freedom on μ.

Single-parameter optimization

In this section, we derive the optimization process in the last section for a single-parameter optimization. We optimize the filters by modifying the parameters Bandσ of the Gaussian for each channel i, following Eq. 4. The aim of the single parameter optimization is to obtain the same camera response (i.e energy balance) for the N filters, while taking into account the substrate absorption and the spectral properties of the illumination. So we search for good settings of B _i or σ _i in the transmission characteristics of filters. We fix the sigma for all bands in the amplitude optimization, and we fix the amplitude in the standard deviation optimization. The response ρ _i is calculated for a set of filter transmissions, f(B ₁,…,B _N ) and f(σ ₁,…,σ _N ), when searching respectively for the amplitudes and sigmas. In these two cases, optimal parameters are found such as \(\rho _{1}=\rho _{2}=\ldots =\rho _{N} = \tilde {C}\), a constant value.

For the analysis, we will consider the case of a camera based on N=[3,5,8,12] bands. We chose a three-band sensor to represent the typical color case, five bands are chosen for multispectral color sensors or colorimeters [44]. The eight bands are selected for comparison with an existing filter realization [3, 7]. Twelve bands are chosen, as they are expected to provide the best spectral reconstruction used in multispectral sensors [43, 45].

Energy balance by changing the magnitude

We change sequentially the magnitude of B _i to obtain energetically balanced filters, whereas the value of σ is set as a constant for all bands during the optimization process.

We use a rescale algorithm. First, initial values of B _i,0 are set to 1. Equation 1 is applied to calculate initial response values ρ _i,0 for which channel responses are not balanced (uncorrected filters). Then, the optimal amplitude parameters are found by dividing the highest value among the set by ρ _i,0. The equation could be written as this:

$$ B_{i}=\frac{\rho_{max}}{\rho_{i,0}}, $$

(6)

After normalization between [0,1], we finally find filter parameters B ₁,…,B _N , for which each camera integrated responses ρ _i share the same energy value \(\rho _{1}=\rho _{2}=\ldots =\rho _{N} = \tilde {C}\).

Figures 6(a), (b), (c) and (d) show the results after optimization under the illumination E for a perfect diffuser (O(λ)=1). In the range of wavelengths considered, σ is fixed and is equal to 48.0 for three filters, 28.8 for five filters, 18.0 for eight filters and 12.0 for twelve filters. For visualization and interpretation, we also show the response curve S(λ) of silicon contained in Fig. 2.

We observe that the magnitude of each filter depends directly on the response of silicon. These results highlight the fact that the silicon response is not uniform. In addition, if we consider a linear model of noise, the difference in amplitude is related to the difference of channel noise. The difference among filters increases as the bandwidth of the filters becomes narrow.

Energy balance by changing the standard deviation

This time, B is set as a constant for all bands. As for the amplitude optimization process, Eq. 1 is applied to calculate ρ _i,0 initial values for each filters. In order to obtain the Gaussian values of σ _i (on a fixed interval) for which the filters share the same energy, we search for the minimum of the single-variable function defined as this:

$$ \begin{array}{cc} \arg\min_{\sigma_{i}} |C-\tilde{C}| \text{ such that} & \qquad \sigma_{min} \leq \sigma_{i} \leq \sigma_{max}, \end{array} $$

(7)

where \(C=\frac {\sum {\rho _{i,0}}}{N}\) and \(\tilde {C}\) is the integration result of camera response when changing sigma in Eq. 4.

The obtained energetically balanced filters are presented in Fig. 7(a), (b), (c) and (d), for a flat reflectance object and illuminant E. The amplitude B _i is fixed and is equal to 1.

We observe that the standard deviation of each filter depends directly on the response of silicon. These results highlight the fact that the silicon response is not uniform. In addition, if we consider a linear model of noise, the difference in standard deviation is related to the difference of channel noise. The difference among filters increases as the bandwidth of the filters becomes narrow.

Multi-parameter constrained optimization

The framework tunes both parameters B and σ, in front of a set of illuminations given by the application, and constrained by the above principles. μ is presented as constant here, but may be embedded easily in the optimization parameters if the application allow freedom on the peak sensitivities. A block diagram of the method is presented in Fig. 8.

given a set of linear inequalities that will be defined in the following sections. STD stands for the standard deviation of the set of computed sensor responses.

Constraints on overlapping

On Fig. 5(a) we observed too much overlapping. To avoid overlapping, according to the Fig. 9(b), we must respect the conditions of Eq. 10. We use the Gaussian percentiles to define this constraint, expressed as an integer weight of the standard deviation.

Equation 11 puts the Eq. 10 in the format A.x≤b in order to prepare a mathematical model for the optimization tool. A is a matrix and b is a vector. The constraint set of equation can be written as this:

$$ \begin{array}{l} \mu_{i-1}-P\sigma_{i-1}\leq\mu_{i}-Q\sigma_{i} \\ \mu_{i-1}+P\sigma_{i-1}\leq\mu_{i}-P\sigma_{i}\\ \mu_{i}+P\sigma_{i}\leq\mu_{i+1}-P\sigma_{i+1}\\ \mu_{i}+Q\sigma_{i}\leq\mu_{i+1}+P\sigma_{i+1}~,\\ \end{array} $$

(10)

$$ \left(\begin{array}{lll} -P & Q & 0 \\ P & P & 0 \\ 0 & P & P \\ 0 & Q & -P \\ \end{array} \right) \left(\begin{array}{lll} \sigma_{i-1} \\ \sigma_{i} \\ \sigma_{i+1} \end{array} \right) \leq \left(\begin{array}{lll} \mu_{i}-\mu_{i-1} \\ \mu_{i}-\mu_{i-1} \\ \mu_{i+1}-\mu_{i} \\ \mu_{i+1}-\mu_{i} \\ \end{array} \right) $$

(11)

where the values P and Q are user parameters. These are defined as 0<P<Q. The value of P can more likely be 1 and the value of Q, 3, according to the energy distribution of a Gaussian, presented on Fig. 9(a).

General results

The code can be downloaded online along with different sensor responses to be tested. The set of constraints was defined as above. The range of wavelength chosen is the visible spectrum (380−780 n m), but the acquisition model described in the methods section can be generalized to NIR wavelengths, as long as the imaging model used is valid, i.e. without fluorescent effects and emission from objects. The set of constraints are linear functions. In practice, the multispectral device can be subject to a limitation of illuminant intensity, and to a limited range of exposure times, which is not addressed here. We also consider the sensor to have a linear response. If it was not the case, a linearization function may be used to correct the data. Figure 10 and Table 1 show a list of standard illuminants used. To perform simulations, we consider the case of a commercial sensor by SONY [47], and use the spectral sensor response presented in Fig. 2. A similar process can be used for most of the sensors, for which the absorption by wavelength is known or measurable.

Illuminant	CCT (K)	Environment
E	-	Theoretical reference
A	2856	Incandescent
F3	3450	White Fluorescent
D50	5003	Horizon Light
D55	5503	Mid-day Light
D65	6504	Standard daylight, sRGB
D75	7504	Daylight in North

These three cases are illustrated in Fig. 11. The first column shows the resulting set of filters for N=3,5,8 and 12, for which the optimization process converges successfully. So the integration for each area curve is quite constant along the wavelengths, and decreases with the number of channels N. We can clearly distinguish, for example, the shape contribution of the sensor sensitivity in Fig. 2 when using for example 12 filters and illuminant E.

The middle column is for the case where the optimization converges and where a minimum standard deviation among the filters is acceptable. Here, the D series of the illuminants, which have a globally homogeneous shape, are used. These are not very far from each other in term of emissivity, and the optimization succeeded.

The third column shows the case where the optimization process fails to recover proper filter shapes within this set of constraints. It is due to the fact that we use too many different viewing illuminants from which the shapes of emissivity are too far from each others. The transmission peaks remain at the minimal value (0.3) and the filters are not optimized since the standard deviation is more than 50% of the energy mean among the bands. Nevertheless, it is important to note that a good convergence could be found if constraints like B _min , P or Q are adjusted and relaxed.

For comparison with a practical case, the response of the real camera design presented in Fig. 1 has a response mean of 3.8255 and a standard deviation of 1.71 through illuminant E. It typically corresponds to the case where the filters are not well optimized energetically.

Specific scenarios

To illustrate our method in practice, we propose to study three specific scenarios.

We study first the case of underwater imaging, where absorption of ocean water [48] is added in the model presented in the methods section. We use a visible range of wavelength (380−780 n m), one projector illuminant (tungsten), and 3 bands. A result of optimization with multiple parameters is shown in Fig. 12(a). We obtain an average score of 2.29 and a standard deviation of 6.74e ⁻⁶. The optimization process succeeds in finding a minimum for the objective function in this case. The ocean water has a noticeable absorption in the red wavelength (between 700 and 780 n m), but we notice that the third "red" filter has a reduced transmission value than the first one. It could be explain by the fact that the tungsten illuminant has a emission even lower in the range of 380−500 n m, which explains why the optimized filters are compensated.

Another option is proposed on Fig. 12(b), where the peak sensitivities have been chosen according to the optimized sensitivities proposed by Parmar et al. [49]. We observe that the sensitivities proposed in this last case is more efficient. Index of quality between two filter banks, such as in Figs. 12(a) and (b), may be to choose the best sensitive option coupled with the best results in resulting color image.

A third case is a typical environment and camera system for colorimetric imaging purpose in a natural environment. This case uses 5 bands and performs well under all D illuminations. By doing the optimization process, we find an average of 3.07 and a standard deviation of 0.42. The corresponding filters are shown in Fig. 12(c). Here, the optimization process converges to an acceptable solution with a certain degree of freedom.

A fourth case example in Fig. 12(d) is the typical multispectral camera scenario for computer vision, where 8 bands in the visible and NIR (380−830 n m) are used. This scenario works with a relatively large variety of illuminants (A, F and D65). We obtain an average score of 0.82 with a standard deviation of 0.60, which is very high and not acceptable. Here, the optimization process fails to find a solution, due to the over-constrained environment. This is due to the selected set of illuminants that shows very different behaviors in term of spectral emissivity.

Also, for these last two cases, we observe low transmittance peak. Even if this order of efficiency magnitude is reasonable in practice according to the manufacturing status, this may be taken into account in releasing the constraints. Also, a larger standard deviation between the energies of the filters may be considered acceptable. It is though not too easy to decide on that before to incorporate noise influence into the simulations.

Discussion

The main contribution of this work is to show that the energy balance of SFA sensors is reachable in several illumination conditions by using an optimization process constrained by practical issues.

What we have not addressed here, is how this influences the actual imaging pipeline in term of noise reduction or in term of application accuracy. Indeed, we demonstrated here that we could manage to avoid saturated values while others would be very noisy low signal. In order to infer the effect on the complete imaging pipeline and fix/relax the constraints more accurately, one may either incorporate noise evaluation or incorporate this methodology into a more comprehensive optimization toward spectral reconstruction, demosaicing or color reproduction. Further work includes then the objective definition of the constraints.

Also, the optical model that we use is rather limited in practice since filter transmittance are not Gaussian. In order to demonstrate our purpose, it was not critical since the FWHM part of the curves are rather similar to real filters, however, when it comes to noise and correlation between channels, it makes no doubt that a more accurate optical transmittance model must be used. We also did not address multi-lobe sensitivities, indeed it could be beneficial for increasing the sensitivity of the sensor, however, the spectral community tends to prefer single-lobes in order to separate better between wavelengths. Such concept would be nevertheless beneficial for color imaging based on less than 4 bands. However, with the increasing of bands, even color will not gain much from multi-lobe curves.

Our main message is that this problem of energy balance, which could be reasonably ignored in the case of CFA or SFA cameras based on large interference filters, may not be ignored in the SFA case.