Acquisition model and balanced sensor
We consider a SFA camera that consists of N spectral channels characterized by F, a set of spectral transmittance filters, such as F={f
1,…,f
N
}. The spectral sensitivity of the sensor without a SFA is represented by S(λ). For a given illuminant l(λ), and an object of reflectance O(λ), we obtain a set of values Π={ρ
1,…,ρ
N
}. Every value ρ
i
is given by filter f
i
, i∈⟦1,N
⟧ such as in Eq. 1.
$$ \rho_{i}=\int_{\Lambda} O(\lambda) l({\lambda}) S(\lambda) f_{i}(\lambda) d\lambda $$
(1)
This equation could also include the transmission of the medium in which the system is used, e.g. underwater imaging.
We define energy balance of a sensor for one illuminant when, for a perfect diffuser, the variance of Π is equal to 0, in other words, when ρ
1=ρ
2=…=ρ
N
. The flat reflectance object may be replaced by a specific reflectance property characterizing the application. For compactness, we denote Π=X(F), a function where X define a vector Π from a set of filters F, according to the imaging model of Eq. 1.
Definition 1
In order to guarantee energy balance, we want to find F among the general set of possible transmittance curves T, such as it minimizes V(Π), the variation among components of Π. If we note Y(F)=V(X(F)), this can be summed up as finding F such as min∀F∈T
Y(F).
To generalize this definition to represent practical illuminant conditions for a range of applications, we introduce a set of illuminants L=[l
1,…,l
M
] assumed to be representative of lightning conditions. This set may be dependent on the application. Then we rewrite Eq. 1 as Eq. 2, with j∈⟦1,M
⟧.
$$ \rho_{i,j}=\int_{\Lambda} O(\lambda) l_{j}{(\lambda)} S(\lambda) f_{i}(\lambda) d\lambda $$
(2)
Definition 2
Again, for a perfect diffuser, we want to find an optimal set F, which minimize the variation among components of Π
′. Π
′ is a set of Π
j
, such as Π
′={Π
1,…,Π
M
}. Similarly, we note Y
′(F)=V(X
′(F)), where X
′ is a set of functions according to Eq. 2 and we can write the problem as finding F such as min∀F∈T
Y
′(F).
The solution where this converges might not be suitable for multispectral acquisition. Indeed, an obvious solution to this problem is when F is composed of the same filters (i.e. f
1=f
2=…f
N
). To ensure the practicality of the filter set, we constrain the set F to a reduced subset of T. The reduced set is comprised of Gaussian filters distributed over the considered part of the spectrum. Then, F is characterized by its set of peak sensitivity locations Ω=μ
1,…,μ
N
, which is fixed by the application. We propose to give a freedom to two parameters in order to find an optimal F using the standard deviation of each Gaussian σ
1,…,σ
N
and the maximum transmittance B
1,…,B
N
. This model could be adapted to several models of transmittance, but for clarity of communication, it is easier to work with Gaussian filters.
According to Definition
1, one may still find a minimum convergent solution. However, the set might not be viable in practice, due to limited efficiency or unpractical width. According to Definition 2, we can not converge to zero due to the variability in illuminations. But we can give a degree of freedom, ε, such as Y
′(F)+ε=0.
In this context, we must put constraints on the model, in order to ensure the practicability of the filters set (and its realization), a reasonable variation among the set and relatively good maximum transmittance. These set of constraints are to be defined in next Sections.
Sensor considerations
A major element in this system is the shape of the sensor spectral response S(λ). The silicon response is a fixed parameter in our methodology because we assume the choice of sensor to be made a priori, based on the application. An example of such a characteristic is presented in Fig. 2.
A photon is characterized by a wavelength λ which is equivalent to an energy E. The inverse relationship between the energy of a photon and the wavelength is defined by:
$$ E=\frac{hc}{\lambda}, $$
(3)
where c is the speed of the light in vacuum and h the Planck’s constant.
Photons with an energy bellow 1.1e
V will go through silicon without interaction [35]. This corresponds, according to Eq. 3, to a theoretical wavelength larger than 1125 n
m. The structure of the sensor will, in fact, reduce this wavelength prior to reaching the theoretical limit [35]. Our model does not include the optical effects. Indeed, the layer of silicon can act as a Fabry-Pérot interferometer [36], which modifies the absorption properties of the substrate. Without the optical effects, we can use directly the spectral response as defined in Fig. 2.
Filter considerations
The filters considered in SFAs are most often a realization of a Fabry-Pérot interferometers today [5, 7, 37]. Future technical development may give rise to nano-tube or nano-hole SFA realizations [10]. Filters can be metallic filters [38, 39] or nano structure filters [40], and the substrate is often silicon or doped silicones, germanium, AsGa, indium, etc.
We use a simple Gaussian model to approximate Fabry-Pérot interferometric filters, such as realized by SILIOS [37] and IMEC [8]. This approximation has been extensively done in the literature, in Fig. 3, we verify the impact of this approximation in term of energy inaccuracy before to push further our description. This figure shows the difference between a Gaussian approximation (G), a model of Fabry-Pérot (FPM), and measurement of a practical Fabry-Pérot designed by SILIOS (FPR). If we compute the differences between these curves as integral of function difference, we obtain the following numbers: d(F
P
R,G)=36.74%, d(F
P
M,F
P
R)=9.96% et d(F
P
M,G)=46.70%. Although the difference between G and the Fabry-Pérot instances seems to indicate that the approximation may not be adequate, the difference between the FPM and the FPR indicates that depending on the technology, general models would not be adequate either due to noise and specific -secret- tuning of the manufacturer. We then make the statement that for illustration purpose it is more interesting to develop a methodology with Gaussian and to let the user of this methodology to tune it toward his own model of filters. In addition, if we only compute the difference in peak values, the Gaussian model is closer to the Fabry-Pérot model than the measurement, due to the realization process. If we consider the differences between these curves within the standard deviation of the Gaussian, i.e. full width at half maximum (FWHM), the difference between FPM and FPG is greater than the difference with G and validate the use of the model by the assumption that the difference comes from the lobes of the Fabry-Pérot filter, which generate only an offset spectrally consistent. This simple computation may also serve in the analysis of literature results obtained with Gaussian simulations.
The shape of the filter according to the Gaussian model depends on three parameters, namely the standard deviation σ, the scaling factor (the peak efficiency) B, and the average of the distribution (the location of the filter in the electromagnetic range) μ. Example of filters are given in Fig. 4 where we show a set of three Gaussian filters uniformly distributed. Within this model, we can write f
i
as in Eq. 4.
$$ \begin{array}{l} f_{i}(\lambda)=B_{i}\frac{1}{\sigma_{i} \sqrt{2\pi }}e^{-\frac{1}{2}\left(\frac{\lambda-\mu_{i}}{\sigma_{i}}\right)^{2}}\\ \end{array} $$
(4)
where μ
i
are fixed parameters according to the application and σ
i
and B
i
correspond to the given degree of freedom. The distribution of μ may be given a degree of freedom, but according to our knowledge, users usually optimize it for a specific application. Here, we consider that μ are provided in a priori design and as example, we consider that it is an equi-distributed set or a list of selected peaks, such as shown in the Results section. In some cases, where μ are not provided or in a more global framework, it may be embedded into the optimization process with little extra work and restrained so that all channels do not end with the same transmittance.
As for example purpose, we define an equi-distributed μ set, within the global applicable range of wavelengths Δ
Λ=λ
max
−λ
min
and the number of channels N, so that the central positions μ
i
of each Gaussian are fixed before the optimization process. Consequently, in this case, the position μ
i
can be defined like in Eq. 5:
$$ \begin{array}{l} \mu_{i}=\lambda_{min}+\frac{\Delta \Lambda(2i-1) }{2N},\\ \end{array} $$
(5)
In a practical realization of an SFA system, we may consider two situations, as shown in Figs. 5(a) and (b).
We can consider two sensor configurations. One is very specific to some applications, such as in medical imaging [41, 42], where a very narrow spectral band of about 10 nm or ratio computed between specific narrow bands are required. In this case, Fig. 5(b) is a typical setup, and peaks cannot be given freedom. Another situation, where very wide bands are considered, does not to provide an optimal results either for demosaicing or for spectral reconstruction. After looking at the results of Sadeghipoor et al. [33] and Wang et al. [43], it seems that rather a relatively wide band is adequate for general demosaicing and spectral reconstruction. Indeed, we want to preserve separability between wavelengths if possible, e.g. NIR channels must not be polluted by visible information for computer vision, while maximizing spectral correlation for demosaicing. In this last case, we may authorize a little freedom on μ.
Single-parameter optimization
In this section, we derive the optimization process in the last section for a single-parameter optimization. We optimize the filters by modifying the parameters Bandσ of the Gaussian for each channel i, following Eq. 4. The aim of the single parameter optimization is to obtain the same camera response (i.e energy balance) for the N filters, while taking into account the substrate absorption and the spectral properties of the illumination. So we search for good settings of B
i
or σ
i
in the transmission characteristics of filters. We fix the sigma for all bands in the amplitude optimization, and we fix the amplitude in the standard deviation optimization. The response ρ
i
is calculated for a set of filter transmissions, f(B
1,…,B
N
) and f(σ
1,…,σ
N
), when searching respectively for the amplitudes and sigmas. In these two cases, optimal parameters are found such as \(\rho _{1}=\rho _{2}=\ldots =\rho _{N} = \tilde {C}\), a constant value.
For the analysis, we will consider the case of a camera based on N=[3,5,8,12] bands. We chose a three-band sensor to represent the typical color case, five bands are chosen for multispectral color sensors or colorimeters [44]. The eight bands are selected for comparison with an existing filter realization [3, 7]. Twelve bands are chosen, as they are expected to provide the best spectral reconstruction used in multispectral sensors [43, 45].
Energy balance by changing the magnitude
We change sequentially the magnitude of B
i
to obtain energetically balanced filters, whereas the value of σ is set as a constant for all bands during the optimization process.
We use a rescale algorithm. First, initial values of B
i,0 are set to 1. Equation 1 is applied to calculate initial response values ρ
i,0 for which channel responses are not balanced (uncorrected filters). Then, the optimal amplitude parameters are found by dividing the highest value among the set by ρ
i,0. The equation could be written as this:
$$ B_{i}=\frac{\rho_{max}}{\rho_{i,0}}, $$
(6)
After normalization between [0,1], we finally find filter parameters B
1,…,B
N
, for which each camera integrated responses ρ
i
share the same energy value \(\rho _{1}=\rho _{2}=\ldots =\rho _{N} = \tilde {C}\).
Figures 6(a), (b), (c) and (d) show the results after optimization under the illumination E for a perfect diffuser (O(λ)=1). In the range of wavelengths considered, σ is fixed and is equal to 48.0 for three filters, 28.8 for five filters, 18.0 for eight filters and 12.0 for twelve filters. For visualization and interpretation, we also show the response curve S(λ) of silicon contained in Fig. 2.
We observe that the magnitude of each filter depends directly on the response of silicon. These results highlight the fact that the silicon response is not uniform. In addition, if we consider a linear model of noise, the difference in amplitude is related to the difference of channel noise. The difference among filters increases as the bandwidth of the filters becomes narrow.
Energy balance by changing the standard deviation
This time, B is set as a constant for all bands. As for the amplitude optimization process, Eq. 1 is applied to calculate ρ
i,0 initial values for each filters. In order to obtain the Gaussian values of σ
i
(on a fixed interval) for which the filters share the same energy, we search for the minimum of the single-variable function defined as this:
$$ \begin{array}{cc} \arg\min_{\sigma_{i}} |C-\tilde{C}| \text{ such that} & \qquad \sigma_{min} \leq \sigma_{i} \leq \sigma_{max}, \end{array} $$
(7)
where \(C=\frac {\sum {\rho _{i,0}}}{N}\) and \(\tilde {C}\) is the integration result of camera response when changing sigma in Eq. 4.
The obtained energetically balanced filters are presented in Fig. 7(a), (b), (c) and (d), for a flat reflectance object and illuminant E. The amplitude B
i
is fixed and is equal to 1.
We observe that the standard deviation of each filter depends directly on the response of silicon. These results highlight the fact that the silicon response is not uniform. In addition, if we consider a linear model of noise, the difference in standard deviation is related to the difference of channel noise. The difference among filters increases as the bandwidth of the filters becomes narrow.
Limitations
When we optimize the magnitude or the standard deviation of the filters, we reach an exact solution in a few iterations, where standard deviation of the values {ρ
1,ρ
2,…,ρ
N
} is near or equal to zero. However, we observe an important reduction of efficiency in most of the optimal camera responses. Indeed, typically, 70% of the energy should be filtered out to guarantee the balance. This result is illustrated in Fig. 6(d). In Figs. 6(c) and (d), we can see that we severely reduced the transmittance efficiency of most of the filters. This appears to be an inefficient situation for practical implementation, considering the noise that might appear with low sensitivity filters. Such optimization is not adequate in practice. Spectral overlapping and spectral gap problems among bands are observed when we perform the optimization on σ (see Figs. 7(c) and (d)). Moreover, when illumination changes, filters change accordingly to the coupled pair {sensor, illuminant}. In a practical case, several illuminations may be considered, and an exact optimization may not give good results in several configurations.
These results suggest that a constrained multi-parameter optimization may be a better fit to realize practical filter banks, which is the subject of the following section.
Multi-parameter constrained optimization
In the last section, the optimization of parameters σ and B were proposed independently, such as presented by Péguillet et al. [29]. In this section, we present a new framework for finding optimal spectral filter responses according to the global energy balance of the camera system. To this aim, we define a set of constraints that should guarantee the following principles:
-
1.
There should be no gap in the spectral domain, otherwise some spectral information will be lost for some wavelengths.
-
2.
There should not be too much spectral overlapping of a filter.
-
3.
The sensor must globally be sensitive. Peak attenuation should be controlled.
-
4.
Energy balance must be achieved for an illumination set.
The framework tunes both parameters B and σ, in front of a set of illuminations given by the application, and constrained by the above principles. μ is presented as constant here, but may be embedded easily in the optimization parameters if the application allow freedom on the peak sensitivities. A block diagram of the method is presented in Fig. 8.
The silicon response S(λ) is a specific parameter as it corresponds to the selected imaging device. The illumination l
j
(λ) is also given as parameter. The global problem formula is as in Eq. 8:
$$ \arg\min_{\sigma_{i}\text{,}B_{i}} \text{STD}(\Pi') $$
(8)
given a set of linear inequalities that will be defined in the following sections. STD stands for the standard deviation of the set of computed sensor responses.
Constraints on amplitude
The value of the amplitude B must be as high as possible, otherwise the signal will be too weak compared to noise. In the same time, B
max
is not necessary 100% transmittance since a limitation may be imposed by the technology used to manufacture the filters. So, we take B
min
and B
max
in Eq. 9 as a minimum and maximum amplitude parameters. In any case, the maximum value can not be over 100%, and we notice that it may be beneficial for one band to be less efficient in order to be balanced with the others.
$$ B_{min} \leq B\leq B_{max} $$
(9)
Constraints on overlapping
On Fig. 5(a) we observed too much overlapping. To avoid overlapping, according to the Fig. 9(b), we must respect the conditions of Eq. 10. We use the Gaussian percentiles to define this constraint, expressed as an integer weight of the standard deviation.
Equation 11 puts the Eq. 10 in the format A.x≤b in order to prepare a mathematical model for the optimization tool. A is a matrix and b is a vector. The constraint set of equation can be written as this:
$$ \begin{array}{l} \mu_{i-1}-P\sigma_{i-1}\leq\mu_{i}-Q\sigma_{i} \\ \mu_{i-1}+P\sigma_{i-1}\leq\mu_{i}-P\sigma_{i}\\ \mu_{i}+P\sigma_{i}\leq\mu_{i+1}-P\sigma_{i+1}\\ \mu_{i}+Q\sigma_{i}\leq\mu_{i+1}+P\sigma_{i+1}~,\\ \end{array} $$
(10)
$$ \left(\begin{array}{lll} -P & Q & 0 \\ P & P & 0 \\ 0 & P & P \\ 0 & Q & -P \\ \end{array} \right) \left(\begin{array}{lll} \sigma_{i-1} \\ \sigma_{i} \\ \sigma_{i+1} \end{array} \right) \leq \left(\begin{array}{lll} \mu_{i}-\mu_{i-1} \\ \mu_{i}-\mu_{i-1} \\ \mu_{i+1}-\mu_{i} \\ \mu_{i+1}-\mu_{i} \\ \end{array} \right) $$
(11)
where the values P and Q are user parameters. These are defined as 0<P<Q. The value of P can more likely be 1 and the value of Q, 3, according to the energy distribution of a Gaussian, presented on Fig. 9(a).
Constraints on gaps
On Fig. 5(b) we observed gaps. In the same manner as for the overlapping constraints, according to the Fig. 9(c), avoiding gap is done by respecting the conditions of Eq. 12. We can then define linear constraints as follow:
$$ \begin{array}{l} \mu_{i-1}+P\sigma_{i-1}\geq\mu_{i}-Q\sigma_{i} \\ \mu_{i-1}+Q\sigma_{i-1}\geq\mu_{i}-P\sigma_{i} \\ \mu_{i}+Q\sigma_{i}\geq\mu_{i+1}-P\sigma_{i+1} \\ \mu_{i}+P\sigma_{i}\geq\mu_{i+1}-Q\sigma_{i+1}~, \end{array} $$
(12)
$$ \left(\begin{array}{lll} -P & -Q & 0 \\ -Q & -P & 0 \\ 0 & -Q & -P \\ 0 & -P & -Q \\ \end{array} \right) \left(\begin{array}{lll} \sigma_{i-1} \\ \sigma_{i} \\ \sigma_{i+1} \end{array} \right) \leq \left(\begin{array}{lll} \mu_{i-1}-\mu_{i} \\ \mu_{i-1}-\mu_{i} \\ \mu_{i}-\mu_{i+1} \\ \mu_{i}-\mu_{i+1} \\ \end{array} \right) $$
(13)
where the values P and Q are the same than the values used with the constraints to avoid the overlapping issue.
Resolution of the inverse problem
The optimization framework was implemented in Matlab using the fmincon routine to solve the function 8. The algorithm used is the active-set. In order to select the optimization solving method, we followed the MATLAB recommendation of use of fmincon function [46]. Indeed, the size of the problem is not very large (we can store all data during optimization, and linear algebra can be easily computed on them), the problem does not seem to show local minimum issues, although several approximated solutions may be possible. So this solution would provide the fastest optimization of parameters. In test, we noticed that other existing algorithms would reach the same results.