Energy balance in Spectral Filter Array camera design

Abstract

Background

Multispectral imaging permits to capture more spectral information on object surface properties than color imaging. This is useful for machine vision applications. Transmittance spectral filter arrays combined with a solid state sensor form an emerging technology used for snapshot acquisition. In spectral filter arrays technology, the sensitivities of the camera have critical consequences, not only on applications, but also in the viability of the system. We discuss how to balance the energy of each channel in single exposure multispectral imaging.

Methods

We propose a methodology to design filters that permits to reach an optimal balance of energy. We apply this method on practical illuminations combined with a Gaussian model of transmittance filters.

Results and discussion

Our results demonstrate that we can optimize energetically the global camera response with few efforts for several cases of illumination environments, for a given sensor and a number of spectral channels. This methodology can be embedded in an application-oriented optimization framework.

Conclusions

This methodology enables achieving a great range of optical design, and it can be embedded in an application-oriented optimization framework.

Background

Multispectral imaging (MSI) is a sensing technique that is employed in different spectrum of wavelengths. When used in the context of typical silicon sensors, it may span the electromagnetic spectrum from ultra-violet to near-infrared (NIR) range, approx. [320-1100] nm. Up to now, multiple sensors or exposures were necessary to record such images [1–3]. The generalization of color filter arrays (CFA) into spectral filter arrays (SFA) [3] permits the use of a single sensor and a single exposure to record a multispectral image. This solution can be encapsulated into a compact and affordable MSI solution, easy to handle, which complies with a typical camera pipeline. The principle of SFA imaging is to combine a single sensor with a filter set. Each spatial position of the array captures only one spectral channel similarly to the Bayer filters [4], and the filter transmission characteristics determine the sensor spectral properties. Several realizations of such setup have been shown recently in the literature [5–7]. An example is provided in Fig. 1. Various research directions have been explored: filter characteristics [8–11], adequate full resolution image processing, e.g. demosaicing [12–17], and other aspects [18, 19].

Fig. 1

a Example of a mozaic element in an SFA imaging system composed of 8 channels. b Corresponding relative spectral characteristics of the camera in the work by Lapray et al. [3, 7] (c). Filters transmittance manufactured by SILIOS Technologies [37]. d Sensor response. e Raw image sample of the MacBeth color checker from the SFA camera. f Zooming on an achromatic patch of the raw image to observe energy balance disparity among all channels

In silicon based solid-state sensor image acquisition, when a photo-diode receives too much radiant energy, the maximum number of electrons per pixel is reached [20]. This refers to saturation, and extra information is lost. In addition, the overloaded photo-site distributes charges into adjacent pixels, generating blooming (especially for CCD technology) [21, 22]. On the other hand, if a photo-diode does not receive sufficient photons, the signal to noise ratio (SNR) of the electrical signal generated is too low compared to the global noise level [23]. Thus, when integration time is fixed, the sensor performance is optimal only within a given range of intensities according to its sensitivity. In a gray-level intensity sensor, pixels have a similar sensitivity and it is relatively easy to set an optimal integration time [24, 25].

However, in the case of a SFA set-up, different sensitivities are present on the same chip and may lead to over- and under-exposed classes of pixels, depending on the scene and on the illumination. This is illustrated by Thomas et al. [7], where the sensitivities of the less-sensitive band and the most-sensitive band (see Fig. 1) show a relative efficiency ratio in the interval [11%, 99%] for a set of practical filters, an ideal viewing illuminant and a perfect diffuser. In such cases, it is more likely that a single optimal integration time shall generate either saturated values or under-exposed pixels, as indicated by results reported in their article. Figure 1(e) shows a raw image captured with an SFA camera. On a zoom on Fig. 1(e), we can observe a severe discrepancy between pixel intensity values for a uniform patch.

One might consider several solutions to solve this problem. One solution would be to set an optimal integration time for each pixel. This would be possible by using a multiple layer technology, see e.g. [26]. To the best of our knowledge, this approach is still in development and remains an expensive solution, which does not yet meet the requirements of a low price SFA. Another solution is to use high dynamic range imaging (HDR) based on multiple exposure, see e.g. [27, 28]. Although the solution may be embedded in a real-time framework, it precludes a single-shot SFA capability. Another solution would be to apply gains to re-balance the under-exposed channel, but the SNR would still remain at a similar level despite denoising potential facilities. One might consider this problem in the sensor design and optimize the camera sensitivities for a particular condition. Such an optimization may be constrained by a specific application or by manufacturing constraints in filter characteristics. Energy balance of SFA has been previously considered by Péguillet et al. [29]. This paper develops such a solution within a constrained optimization framework.

Optimization of spectral filters have been extensively studied in the color imaging field for either accurate color acquisition or reproduction, by e.g. Vhrel [30, 31]. Sharma et al. [32] addressed the problem of obtaining a good perceived colorimetric information, under several viewing illuminants. A set of 3 to 7 filters have been designed using their optimization framework, and a linear minimum mean squared error estimator. Simulation shows that their technique performs better and with an improved SNR than the solution initiated by Vhrel [31]. They concluded that the use of 4 bands seems to be the optimal solution to get a good accuracy when recording color under multi-illuminants. This conclusion is shared by Sadeghipoor et al. [33] who proposed an algorithm to optimize the spectral sensitivity function of the camera by finding the best demosaicing matrix for color acquisition. The constraint imposed is that the resulting function contains smooth practical filters, which are assumed to be physically realizable. Filters have been also optimized by Monno et al. [34] in the context of SFAs in order to provide the best demosaicing results. In their study, they used Gaussian models to approximate sensor sensitivities.

In this work, we propose to base on these previous works and to develop a methodology that provides the best filter bank for a single-shot SFA camera, while considering energy balance. The main contribution is in the definition of what is energy balance and how this may be handled within a more comprehensive optimization process that includes higher level information processing (such as demosaicing, spectral signature indicators or spectral reconstruction).

We use a Gaussian model to approximate transmittance filters and to provide simulated examples. The methodology can be extended to any optical model that can describe the transmittance of filters in a few meaningful parameters, e.g. hyper-Gaussians or multi-lobe single cavity Fabry-Pérot, as suggested by one of the anonymous reviewer. The proposed methodology is based on an imaging model and on an empirical set of constraints such as:

• 1. We must avoid over-exposed and under-exposed areas when shooting a scene with the multispectral camera.

• 2. We must guarantee the best post-processing efficiency (e.g demosaicing or spectral reconstruction) by homogenizing independent channel noise and preserving the design of the filter according to the application.

• 3. We want to guarantee system versatility over a set of viewing illuminants.

Methods

Acquisition model and balanced sensor

We consider a SFA camera that consists of N spectral channels characterized by F, a set of spectral transmittance filters, such as F={f ₁,…,f _N }. The spectral sensitivity of the sensor without a SFA is represented by S(λ). For a given illuminant l(λ), and an object of reflectance O(λ), we obtain a set of values Π={ρ ₁,…,ρ _N }. Every value ρ _i is given by filter f _i , i∈⟦1,N ⟧ such as in Eq. 1.

$$ \rho_{i}=\int_{\Lambda} O(\lambda) l({\lambda}) S(\lambda) f_{i}(\lambda) d\lambda $$

(1)

This equation could also include the transmission of the medium in which the system is used, e.g. underwater imaging.

We define energy balance of a sensor for one illuminant when, for a perfect diffuser, the variance of Π is equal to 0, in other words, when ρ ₁=ρ ₂=…=ρ _N . The flat reflectance object may be replaced by a specific reflectance property characterizing the application. For compactness, we denote Π=X(F), a function where X define a vector Π from a set of filters F, according to the imaging model of Eq. 1.

Definition 1

In order to guarantee energy balance, we want to find F among the general set of possible transmittance curves T, such as it minimizes V(Π), the variation among components of Π. If we note Y(F)=V(X(F)), this can be summed up as finding F such as min∀F∈T Y(F).

To generalize this definition to represent practical illuminant conditions for a range of applications, we introduce a set of illuminants L=[l ₁,…,l _M ] assumed to be representative of lightning conditions. This set may be dependent on the application. Then we rewrite Eq. 1 as Eq. 2, with j∈⟦1,M ⟧.

$$ \rho_{i,j}=\int_{\Lambda} O(\lambda) l_{j}{(\lambda)} S(\lambda) f_{i}(\lambda) d\lambda $$

(2)

Definition 2

Again, for a perfect diffuser, we want to find an optimal set F, which minimize the variation among components of Π ^′. Π ^′ is a set of Π _j , such as Π ^′={Π ₁,…,Π _M }. Similarly, we note Y ^′(F)=V(X ^′(F)), where X ^′ is a set of functions according to Eq. 2 and we can write the problem as finding F such as min∀F∈T Y ^′(F).

The solution where this converges might not be suitable for multispectral acquisition. Indeed, an obvious solution to this problem is when F is composed of the same filters (i.e. f ₁=f ₂=…f _N ). To ensure the practicality of the filter set, we constrain the set F to a reduced subset of T. The reduced set is comprised of Gaussian filters distributed over the considered part of the spectrum. Then, F is characterized by its set of peak sensitivity locations Ω=μ ₁,…,μ _N , which is fixed by the application. We propose to give a freedom to two parameters in order to find an optimal F using the standard deviation of each Gaussian σ ₁,…,σ _N and the maximum transmittance B ₁,…,B _N . This model could be adapted to several models of transmittance, but for clarity of communication, it is easier to work with Gaussian filters.

According to Definition 1, one may still find a minimum convergent solution. However, the set might not be viable in practice, due to limited efficiency or unpractical width. According to Definition 2, we can not converge to zero due to the variability in illuminations. But we can give a degree of freedom, ε, such as Y ^′(F)+ε=0.

In this context, we must put constraints on the model, in order to ensure the practicability of the filters set (and its realization), a reasonable variation among the set and relatively good maximum transmittance. These set of constraints are to be defined in next Sections.

Sensor considerations

A major element in this system is the shape of the sensor spectral response S(λ). The silicon response is a fixed parameter in our methodology because we assume the choice of sensor to be made a priori, based on the application. An example of such a characteristic is presented in Fig. 2.

Fig. 2

The spectral response S(λ) (380 to 1100 n m) of the SONY IMX174 sensor, as measured in the work from Point Grey™ [47]

A photon is characterized by a wavelength λ which is equivalent to an energy E. The inverse relationship between the energy of a photon and the wavelength is defined by:

$$ E=\frac{hc}{\lambda}, $$

(3)

where c is the speed of the light in vacuum and h the Planck’s constant.

Photons with an energy bellow 1.1e V will go through silicon without interaction [35]. This corresponds, according to Eq. 3, to a theoretical wavelength larger than 1125 n m. The structure of the sensor will, in fact, reduce this wavelength prior to reaching the theoretical limit [35]. Our model does not include the optical effects. Indeed, the layer of silicon can act as a Fabry-Pérot interferometer [36], which modifies the absorption properties of the substrate. Without the optical effects, we can use directly the spectral response as defined in Fig. 2.

Filter considerations

The filters considered in SFAs are most often a realization of a Fabry-Pérot interferometers today [5, 7, 37]. Future technical development may give rise to nano-tube or nano-hole SFA realizations [10]. Filters can be metallic filters [38, 39] or nano structure filters [40], and the substrate is often silicon or doped silicones, germanium, AsGa, indium, etc.

We use a simple Gaussian model to approximate Fabry-Pérot interferometric filters, such as realized by SILIOS [37] and IMEC [8]. This approximation has been extensively done in the literature, in Fig. 3, we verify the impact of this approximation in term of energy inaccuracy before to push further our description. This figure shows the difference between a Gaussian approximation (G), a model of Fabry-Pérot (FPM), and measurement of a practical Fabry-Pérot designed by SILIOS (FPR). If we compute the differences between these curves as integral of function difference, we obtain the following numbers: d(F P R,G)=36.74%, d(F P M,F P R)=9.96% et d(F P M,G)=46.70%. Although the difference between G and the Fabry-Pérot instances seems to indicate that the approximation may not be adequate, the difference between the FPM and the FPR indicates that depending on the technology, general models would not be adequate either due to noise and specific -secret- tuning of the manufacturer. We then make the statement that for illustration purpose it is more interesting to develop a methodology with Gaussian and to let the user of this methodology to tune it toward his own model of filters. In addition, if we only compute the difference in peak values, the Gaussian model is closer to the Fabry-Pérot model than the measurement, due to the realization process. If we consider the differences between these curves within the standard deviation of the Gaussian, i.e. full width at half maximum (FWHM), the difference between FPM and FPG is greater than the difference with G and validate the use of the model by the assumption that the difference comes from the lobes of the Fabry-Pérot filter, which generate only an offset spectrally consistent. This simple computation may also serve in the analysis of literature results obtained with Gaussian simulations.

Fig. 3

Comparison between Fabry-Pérot filter model (FPM), gaussian model (G) and practical realization of a filter (P4) from [7] (FPG). The differences between these curves as integral of function difference are d(F P R,G)=36.74%, d(F P M,F P R)=9.96% and d(F P M,G) = 46.70%. The difference at FWHM, then d _σ (F P R,G)=−1.05%, d _σ (F P M,F P R)=−1.99% and d _σ (F P M,G)=−3.04%

The shape of the filter according to the Gaussian model depends on three parameters, namely the standard deviation σ, the scaling factor (the peak efficiency) B, and the average of the distribution (the location of the filter in the electromagnetic range) μ. Example of filters are given in Fig. 4 where we show a set of three Gaussian filters uniformly distributed. Within this model, we can write f _i as in Eq. 4.

Fig. 4

A set of Gaussian filters where N=3, $\mu _{i}=\lambda _{min}+\frac {\Delta \Lambda (2i-1) }{2N}$, $\sigma = \frac {\Delta \Lambda }{N \times 4}$ and B _i =1. The gaussians are spaced by 4σ and the intersection between two is at ±2σ

$$ \begin{array}{l} f_{i}(\lambda)=B_{i}\frac{1}{\sigma_{i} \sqrt{2\pi }}e^{-\frac{1}{2}\left(\frac{\lambda-\mu_{i}}{\sigma_{i}}\right)^{2}}\\ \end{array} $$

(4)

where μ _i are fixed parameters according to the application and σ _i and B _i correspond to the given degree of freedom. The distribution of μ may be given a degree of freedom, but according to our knowledge, users usually optimize it for a specific application. Here, we consider that μ are provided in a priori design and as example, we consider that it is an equi-distributed set or a list of selected peaks, such as shown in the Results section. In some cases, where μ are not provided or in a more global framework, it may be embedded into the optimization process with little extra work and restrained so that all channels do not end with the same transmittance.

As for example purpose, we define an equi-distributed μ set, within the global applicable range of wavelengths Δ Λ=λ _max −λ _min and the number of channels N, so that the central positions μ _i of each Gaussian are fixed before the optimization process. Consequently, in this case, the position μ _i can be defined like in Eq. 5:

$$ \begin{array}{l} \mu_{i}=\lambda_{min}+\frac{\Delta \Lambda(2i-1) }{2N},\\ \end{array} $$

(5)

In a practical realization of an SFA system, we may consider two situations, as shown in Figs. 5(a) and (b).

Fig. 5

a First situation: Large overlapping area between the Gaussian filters ($\sigma = \frac {\Delta \Lambda }{7}$). b Second situation: Large gap area between the Gaussian filters ($\sigma = \frac {\Delta \Lambda }{24}$)

We can consider two sensor configurations. One is very specific to some applications, such as in medical imaging [41, 42], where a very narrow spectral band of about 10 nm or ratio computed between specific narrow bands are required. In this case, Fig. 5(b) is a typical setup, and peaks cannot be given freedom. Another situation, where very wide bands are considered, does not to provide an optimal results either for demosaicing or for spectral reconstruction. After looking at the results of Sadeghipoor et al. [33] and Wang et al. [43], it seems that rather a relatively wide band is adequate for general demosaicing and spectral reconstruction. Indeed, we want to preserve separability between wavelengths if possible, e.g. NIR channels must not be polluted by visible information for computer vision, while maximizing spectral correlation for demosaicing. In this last case, we may authorize a little freedom on μ.

Single-parameter optimization

In this section, we derive the optimization process in the last section for a single-parameter optimization. We optimize the filters by modifying the parameters Bandσ of the Gaussian for each channel i, following Eq. 4. The aim of the single parameter optimization is to obtain the same camera response (i.e energy balance) for the N filters, while taking into account the substrate absorption and the spectral properties of the illumination. So we search for good settings of B _i or σ _i in the transmission characteristics of filters. We fix the sigma for all bands in the amplitude optimization, and we fix the amplitude in the standard deviation optimization. The response ρ _i is calculated for a set of filter transmissions, f(B ₁,…,B _N ) and f(σ ₁,…,σ _N ), when searching respectively for the amplitudes and sigmas. In these two cases, optimal parameters are found such as $\rho _{1}=\rho _{2}=\ldots =\rho _{N} = \tilde {C}$, a constant value.

For the analysis, we will consider the case of a camera based on N=[3,5,8,12] bands. We chose a three-band sensor to represent the typical color case, five bands are chosen for multispectral color sensors or colorimeters [44]. The eight bands are selected for comparison with an existing filter realization [3, 7]. Twelve bands are chosen, as they are expected to provide the best spectral reconstruction used in multispectral sensors [43, 45].

Energy balance by changing the magnitude

We change sequentially the magnitude of B _i to obtain energetically balanced filters, whereas the value of σ is set as a constant for all bands during the optimization process.

We use a rescale algorithm. First, initial values of B _i,0 are set to 1. Equation 1 is applied to calculate initial response values ρ _i,0 for which channel responses are not balanced (uncorrected filters). Then, the optimal amplitude parameters are found by dividing the highest value among the set by ρ _i,0. The equation could be written as this:

$$ B_{i}=\frac{\rho_{max}}{\rho_{i,0}}, $$

(6)

After normalization between [0,1], we finally find filter parameters B ₁,…,B _N , for which each camera integrated responses ρ _i share the same energy value $\rho _{1}=\rho _{2}=\ldots =\rho _{N} = \tilde {C}$.

Figures 6(a), (b), (c) and (d) show the results after optimization under the illumination E for a perfect diffuser (O(λ)=1). In the range of wavelengths considered, σ is fixed and is equal to 48.0 for three filters, 28.8 for five filters, 18.0 for eight filters and 12.0 for twelve filters. For visualization and interpretation, we also show the response curve S(λ) of silicon contained in Fig. 2.

Fig. 6

Optimal filters according to the variation of magnitude for N = 3 (a), 5 (b), 8 (c) and 12 (d). This is the optimization result for an illuminant E and a lambertian reflector. The silicon efficiency of the IMX174 SONY sensor used is shown in blue in background

We observe that the magnitude of each filter depends directly on the response of silicon. These results highlight the fact that the silicon response is not uniform. In addition, if we consider a linear model of noise, the difference in amplitude is related to the difference of channel noise. The difference among filters increases as the bandwidth of the filters becomes narrow.

Energy balance by changing the standard deviation

This time, B is set as a constant for all bands. As for the amplitude optimization process, Eq. 1 is applied to calculate ρ _i,0 initial values for each filters. In order to obtain the Gaussian values of σ _i (on a fixed interval) for which the filters share the same energy, we search for the minimum of the single-variable function defined as this:

$$ \begin{array}{cc} \arg\min_{\sigma_{i}} |C-\tilde{C}| \text{ such that} & \qquad \sigma_{min} \leq \sigma_{i} \leq \sigma_{max}, \end{array} $$

(7)

where $C=\frac {\sum {\rho _{i,0}}}{N}$ and $\tilde {C}$ is the integration result of camera response when changing sigma in Eq. 4.

The obtained energetically balanced filters are presented in Fig. 7(a), (b), (c) and (d), for a flat reflectance object and illuminant E. The amplitude B _i is fixed and is equal to 1.

Fig. 7

Optimal filters according to Eq. 7, with the variation of standard deviations σ _i for N = 3 (a), 5 (b), 8 (c) and 12 (d) such as 0.1≤σ _i ≤100. Illuminant E and a lambertian reflector are used. The sensor silicon efficiency is shown in background

We observe that the standard deviation of each filter depends directly on the response of silicon. These results highlight the fact that the silicon response is not uniform. In addition, if we consider a linear model of noise, the difference in standard deviation is related to the difference of channel noise. The difference among filters increases as the bandwidth of the filters becomes narrow.

Limitations

When we optimize the magnitude or the standard deviation of the filters, we reach an exact solution in a few iterations, where standard deviation of the values {ρ ₁,ρ ₂,…,ρ _N } is near or equal to zero. However, we observe an important reduction of efficiency in most of the optimal camera responses. Indeed, typically, 70% of the energy should be filtered out to guarantee the balance. This result is illustrated in Fig. 6(d). In Figs. 6(c) and (d), we can see that we severely reduced the transmittance efficiency of most of the filters. This appears to be an inefficient situation for practical implementation, considering the noise that might appear with low sensitivity filters. Such optimization is not adequate in practice. Spectral overlapping and spectral gap problems among bands are observed when we perform the optimization on σ (see Figs. 7(c) and (d)). Moreover, when illumination changes, filters change accordingly to the coupled pair {sensor, illuminant}. In a practical case, several illuminations may be considered, and an exact optimization may not give good results in several configurations.

These results suggest that a constrained multi-parameter optimization may be a better fit to realize practical filter banks, which is the subject of the following section.

Multi-parameter constrained optimization

In the last section, the optimization of parameters σ and B were proposed independently, such as presented by Péguillet et al. [29]. In this section, we present a new framework for finding optimal spectral filter responses according to the global energy balance of the camera system. To this aim, we define a set of constraints that should guarantee the following principles:

• 1. There should be no gap in the spectral domain, otherwise some spectral information will be lost for some wavelengths.

• 2. There should not be too much spectral overlapping of a filter.

• 3. The sensor must globally be sensitive. Peak attenuation should be controlled.

• 4. Energy balance must be achieved for an illumination set.

The framework tunes both parameters B and σ, in front of a set of illuminations given by the application, and constrained by the above principles. μ is presented as constant here, but may be embedded easily in the optimization parameters if the application allow freedom on the peak sensitivities. A block diagram of the method is presented in Fig. 8.

Fig. 8

Block diagram of the global optimization framework

The silicon response S(λ) is a specific parameter as it corresponds to the selected imaging device. The illumination l _j (λ) is also given as parameter. The global problem formula is as in Eq. 8:

$$ \arg\min_{\sigma_{i}\text{,}B_{i}} \text{STD}(\Pi') $$

(8)

given a set of linear inequalities that will be defined in the following sections. STD stands for the standard deviation of the set of computed sensor responses.

Constraints on amplitude

The value of the amplitude B must be as high as possible, otherwise the signal will be too weak compared to noise. In the same time, B _max is not necessary 100% transmittance since a limitation may be imposed by the technology used to manufacture the filters. So, we take B _min and B _max in Eq. 9 as a minimum and maximum amplitude parameters. In any case, the maximum value can not be over 100%, and we notice that it may be beneficial for one band to be less efficient in order to be balanced with the others.

$$ B_{min} \leq B\leq B_{max} $$

(9)

Constraints on overlapping

On Fig. 5(a) we observed too much overlapping. To avoid overlapping, according to the Fig. 9(b), we must respect the conditions of Eq. 10. We use the Gaussian percentiles to define this constraint, expressed as an integer weight of the standard deviation.

Fig. 9

a Values of percentile quantities related to a Gaussian function and sigma [50]. Constraints to avoid overlapping (b) and gaps (c), accordingly to Eqs. 10 and 12, using P=1 and Q=3

Equation 11 puts the Eq. 10 in the format A.x≤b in order to prepare a mathematical model for the optimization tool. A is a matrix and b is a vector. The constraint set of equation can be written as this:

$$ \begin{array}{l} \mu_{i-1}-P\sigma_{i-1}\leq\mu_{i}-Q\sigma_{i} \\ \mu_{i-1}+P\sigma_{i-1}\leq\mu_{i}-P\sigma_{i}\\ \mu_{i}+P\sigma_{i}\leq\mu_{i+1}-P\sigma_{i+1}\\ \mu_{i}+Q\sigma_{i}\leq\mu_{i+1}+P\sigma_{i+1}~,\\ \end{array} $$

(10)

$$ \left(\begin{array}{lll} -P & Q & 0 \\ P & P & 0 \\ 0 & P & P \\ 0 & Q & -P \\ \end{array} \right) \left(\begin{array}{lll} \sigma_{i-1} \\ \sigma_{i} \\ \sigma_{i+1} \end{array} \right) \leq \left(\begin{array}{lll} \mu_{i}-\mu_{i-1} \\ \mu_{i}-\mu_{i-1} \\ \mu_{i+1}-\mu_{i} \\ \mu_{i+1}-\mu_{i} \\ \end{array} \right) $$

(11)

where the values P and Q are user parameters. These are defined as 0<P<Q. The value of P can more likely be 1 and the value of Q, 3, according to the energy distribution of a Gaussian, presented on Fig. 9(a).

Constraints on gaps

On Fig. 5(b) we observed gaps. In the same manner as for the overlapping constraints, according to the Fig. 9(c), avoiding gap is done by respecting the conditions of Eq. 12. We can then define linear constraints as follow:

$$ \begin{array}{l} \mu_{i-1}+P\sigma_{i-1}\geq\mu_{i}-Q\sigma_{i} \\ \mu_{i-1}+Q\sigma_{i-1}\geq\mu_{i}-P\sigma_{i} \\ \mu_{i}+Q\sigma_{i}\geq\mu_{i+1}-P\sigma_{i+1} \\ \mu_{i}+P\sigma_{i}\geq\mu_{i+1}-Q\sigma_{i+1}~, \end{array} $$

(12)

$$ \left(\begin{array}{lll} -P & -Q & 0 \\ -Q & -P & 0 \\ 0 & -Q & -P \\ 0 & -P & -Q \\ \end{array} \right) \left(\begin{array}{lll} \sigma_{i-1} \\ \sigma_{i} \\ \sigma_{i+1} \end{array} \right) \leq \left(\begin{array}{lll} \mu_{i-1}-\mu_{i} \\ \mu_{i-1}-\mu_{i} \\ \mu_{i}-\mu_{i+1} \\ \mu_{i}-\mu_{i+1} \\ \end{array} \right) $$

(13)

where the values P and Q are the same than the values used with the constraints to avoid the overlapping issue.

Resolution of the inverse problem

The optimization framework was implemented in Matlab using the fmincon routine to solve the function 8. The algorithm used is the active-set. In order to select the optimization solving method, we followed the MATLAB recommendation of use of fmincon function [46]. Indeed, the size of the problem is not very large (we can store all data during optimization, and linear algebra can be easily computed on them), the problem does not seem to show local minimum issues, although several approximated solutions may be possible. So this solution would provide the fastest optimization of parameters. In test, we noticed that other existing algorithms would reach the same results.

Results and discussion

General results

The code can be downloaded online along with different sensor responses to be tested. The set of constraints was defined as above. The range of wavelength chosen is the visible spectrum (380−780 n m), but the acquisition model described in the methods section can be generalized to NIR wavelengths, as long as the imaging model used is valid, i.e. without fluorescent effects and emission from objects. The set of constraints are linear functions. In practice, the multispectral device can be subject to a limitation of illuminant intensity, and to a limited range of exposure times, which is not addressed here. We also consider the sensor to have a linear response. If it was not the case, a linearization function may be used to correct the data. Figure 10 and Table 1 show a list of standard illuminants used. To perform simulations, we consider the case of a commercial sensor by SONY [47], and use the spectral sensor response presented in Fig. 2. A similar process can be used for most of the sensors, for which the absorption by wavelength is known or measurable.

Fig. 10

Illuminants used in the experiments. Corresponding emission spectra between 380 and 780 n m

Table 1 List of relative illuminants used in the experiments and there Correlated Color Temperature (CCT)

Table 2 shows several iterations of the optimization process, for different illuminants and number of channels. By analyzing the results, we distinguish 3 specific behaviors:

• 1. The optimization process converges to a feasible solution s t d(Π ^′)≃0 (highlighted in blue in Table 2). See the model at the beggining of the document.

  Table 2 Optimization for different illuminant sets. Constraint parameters, in common for all optimization results, are: B _min =0.3, B _max =1.0, P=1 and Q=3

• 2. The optimization process converges to an acceptable solution with a certain degree of freedom/spectral response deviation (highlighted in green in Table 2).

• 3. The solution does not converge to an acceptable solution in term of standard deviation among the filter set (highlighted in red in Table 2).

These three cases are illustrated in Fig. 11. The first column shows the resulting set of filters for N=3,5,8 and 12, for which the optimization process converges successfully. So the integration for each area curve is quite constant along the wavelengths, and decreases with the number of channels N. We can clearly distinguish, for example, the shape contribution of the sensor sensitivity in Fig. 2 when using for example 12 filters and illuminant E.

Fig. 11

General results under illuminant E, [D50,D55,D65,D75] and [D65,A,E] (from left to right). Parameters in common for all optimizations: B _min =0.3, B _max =1.0, P=1 and Q=3. Corresponding response means and standard deviation are shown in Table 2

The middle column is for the case where the optimization converges and where a minimum standard deviation among the filters is acceptable. Here, the D series of the illuminants, which have a globally homogeneous shape, are used. These are not very far from each other in term of emissivity, and the optimization succeeded.

The third column shows the case where the optimization process fails to recover proper filter shapes within this set of constraints. It is due to the fact that we use too many different viewing illuminants from which the shapes of emissivity are too far from each others. The transmission peaks remain at the minimal value (0.3) and the filters are not optimized since the standard deviation is more than 50% of the energy mean among the bands. Nevertheless, it is important to note that a good convergence could be found if constraints like B _min , P or Q are adjusted and relaxed.

For comparison with a practical case, the response of the real camera design presented in Fig. 1 has a response mean of 3.8255 and a standard deviation of 1.71 through illuminant E. It typically corresponds to the case where the filters are not well optimized energetically.

Specific scenarios

To illustrate our method in practice, we propose to study three specific scenarios.

We study first the case of underwater imaging, where absorption of ocean water [48] is added in the model presented in the methods section. We use a visible range of wavelength (380−780 n m), one projector illuminant (tungsten), and 3 bands. A result of optimization with multiple parameters is shown in Fig. 12(a). We obtain an average score of 2.29 and a standard deviation of 6.74e ⁻⁶. The optimization process succeeds in finding a minimum for the objective function in this case. The ocean water has a noticeable absorption in the red wavelength (between 700 and 780 n m), but we notice that the third "red" filter has a reduced transmission value than the first one. It could be explain by the fact that the tungsten illuminant has a emission even lower in the range of 380−500 n m, which explains why the optimized filters are compensated.

Fig. 12

Specific scenarios. a Underwater typical environment with evenly-spaced peak centers. b Underwater typical environment with manual peak center selection at 440, 540 and 590 n m, chosen such as in the optimized color camera described in [49]. c Typical colorimetric camera setup for color imaging. Illuminants used: D50-D55-D65-D75. d Typical multispectral camera for computer vision. Illuminants used: A-D65-F3

Another option is proposed on Fig. 12(b), where the peak sensitivities have been chosen according to the optimized sensitivities proposed by Parmar et al. [49]. We observe that the sensitivities proposed in this last case is more efficient. Index of quality between two filter banks, such as in Figs. 12(a) and (b), may be to choose the best sensitive option coupled with the best results in resulting color image.

A third case is a typical environment and camera system for colorimetric imaging purpose in a natural environment. This case uses 5 bands and performs well under all D illuminations. By doing the optimization process, we find an average of 3.07 and a standard deviation of 0.42. The corresponding filters are shown in Fig. 12(c). Here, the optimization process converges to an acceptable solution with a certain degree of freedom.

A fourth case example in Fig. 12(d) is the typical multispectral camera scenario for computer vision, where 8 bands in the visible and NIR (380−830 n m) are used. This scenario works with a relatively large variety of illuminants (A, F and D65). We obtain an average score of 0.82 with a standard deviation of 0.60, which is very high and not acceptable. Here, the optimization process fails to find a solution, due to the over-constrained environment. This is due to the selected set of illuminants that shows very different behaviors in term of spectral emissivity.

Also, for these last two cases, we observe low transmittance peak. Even if this order of efficiency magnitude is reasonable in practice according to the manufacturing status, this may be taken into account in releasing the constraints. Also, a larger standard deviation between the energies of the filters may be considered acceptable. It is though not too easy to decide on that before to incorporate noise influence into the simulations.

Discussion

The main contribution of this work is to show that the energy balance of SFA sensors is reachable in several illumination conditions by using an optimization process constrained by practical issues.

What we have not addressed here, is how this influences the actual imaging pipeline in term of noise reduction or in term of application accuracy. Indeed, we demonstrated here that we could manage to avoid saturated values while others would be very noisy low signal. In order to infer the effect on the complete imaging pipeline and fix/relax the constraints more accurately, one may either incorporate noise evaluation or incorporate this methodology into a more comprehensive optimization toward spectral reconstruction, demosaicing or color reproduction. Further work includes then the objective definition of the constraints.

Also, the optical model that we use is rather limited in practice since filter transmittance are not Gaussian. In order to demonstrate our purpose, it was not critical since the FWHM part of the curves are rather similar to real filters, however, when it comes to noise and correlation between channels, it makes no doubt that a more accurate optical transmittance model must be used. We also did not address multi-lobe sensitivities, indeed it could be beneficial for increasing the sensitivity of the sensor, however, the spectral community tends to prefer single-lobes in order to separate better between wavelengths. Such concept would be nevertheless beneficial for color imaging based on less than 4 bands. However, with the increasing of bands, even color will not gain much from multi-lobe curves.

Our main message is that this problem of energy balance, which could be reasonably ignored in the case of CFA or SFA cameras based on large interference filters, may not be ignored in the SFA case.

Conclusions

We present a framework that permits filter optimization that achieves energy balance in SFA device configurations. This method can be useful for a great range of applications, where multiple captures, noise or dynamic range are crucial aspects in the final response of the camera. The implementation permits an investigation on how constraints imposed on the system can affect the efficiency and the reliability of the final product under several illumination and capture conditions. Various simulations have been shown to test robustness of the proposed method. Further work envisions the incorporation of this methodology into the application dependent or general purpose filter optimization, its implication into the imaging pipeline and practical realizations of filter and practical evaluation of SFA cameras.