Extracting Super-resolution Structures inside a Single Molecule or Overlapped Molecules from One Blurred Image

1 / 24 Extracting Super -re solution Structu res inside a Single M olecule or Overlapped Mole cules fr om One Blurr ed Im age Y aohua Xie * ( http://orcid.org /0000-0001-6780-3156 ) Y aohua.Xie@hotmail.com , or Edward.Y .Sheffield@hotmail.com Abstract: In some super -resolution techniques, adjacent points are illuminated at different ti mes. Thereby, their locations and li ght intensities can be detected even if th e images are ver y blurred due to diffraction. According to conventional theori es, the points’ inner details cannot be recovered because the images’ high fr equenc y components are removed due to the diff raction -limit. But this study finds an exception, and full in formation can be ex tracted from a diffraction-blurred image. In such a “resolvable condition”, neither profile nor detail information is damaged b y di ffraction. Thereby, it can be recovered reversibl y by solving equation systems in spatial domain or frequenc y domain. This condition is tig htly relevant to the imag ing condition of existing super-resolution techniques. Based on the condition, a method is proposed which ca n achieve unlimited high resolutions in principle, and its effectiveness is demonstr ated by both theoretical analysis and simulation experiments. It can also work without any observed image outside the region of interest. Simulation experiments also show its tolerance to certain level of noise. Keywords: Super-resolution; diffraction-limit; resolvable condition; isolated lighting; positive effective PSF; equation sy stem 1. Introduction When an object (sample) is im aged b y a conventional light microscope, the r esult is not an ideal image which shows sharp details. I nstead, it is equivalent to the ideal image convolved with a P oint Spread Function (PSF) whose central part is cal led Airy disk. Therefore, even a point is infinitel y small, its imag e is an Airy-disk-shaped pattern rather than an ideal point. I n 1873, Ernst Abbe first described the diffraction-limi t: for two points wi th a distance less than a ha lf of visible light’s wavelength, i.e., about 200~300nm, their imag es overlap each other and cannot be resolved. Usually, the Ra y leigh Cr iterion can be adopted to judge wheth er points are resolvable. Samples’ structures smaller than th e diffraction- limit were not re solvable with suc h microscopes unti l super- resolution techniques emerged. These techniques are mainly divided int o t wo categories [ 1]. The first categor y uses struct ural-illumination to image the sample multiple times, and then process es the resulting images to get a super-resolution image. Representative t echniques: STED [2] , RESOLFT [3], S I M [ 4], NL-SIM [5], et al. The s econd category manages to turn on individual molecules at di fferent times, i.e., separates them by time, and then also reconstructs a super- 2 / 24 resolution image. Rep resentative techniques: PA LM [6], S TORM [ 7,] PAINT [8] , et al. Besides, a technique named MINFLUX [9] combines the advantages of the two categories. It can localize individual mol ecules with ultra-high pr ecision. There is a different technique named Ex pansion Microscopy (ExM) [ 10]. I t ex pands samples physically to resolve structures which are unresolvable directl y. Besides, there are some other t y pes of super-resolution technique, e.g., fluctuation-based and comput er -vision-based approaches [11-13]. By now, super-resolution techniques have not only break the diffraction-limit, but also improved resolutions significantly. In techniques such as STED, PA L M, STORM, Confoca l, etc., lum inous points are distant from one another (or there might be only one luminous point at one time). The ir images (almost) do not overlap, thereby the locations and light intensities of these points can be extracted from the blurre d image. Inspired b y these techniques, we find a “ resolvable condition ” (please refer to METHODS for its definition) relevant to their imaging conditions. In the “resolvable condition ” , structures (both inter- points and inn er -points) smaller than the diffraction- limit can be extracted directly from the blurred image, even if the points within the structures are imaged at the same time. From the point view of fre quenc y , the image’s high frequenc y p art is filtered out by the m icroscope. But the structures’ full information (inc luding both profile and details) could still be rec overed from the low frequenc y part, in the “resolvable condition ” . Earl y researches h ave discusse d some mathematic theory for recovering signals from low pass data [14] , while this stud y finds tha t the detail information is not damaged b y diffraction, in the “r esolvable condition ”; then we proposed a technique to recover a diffraction-blurred ima ge ’s Region of Interest (ROI) under this condition, in spatial domain and in frequency domain respectively. 2. Methods 2.1. Background analysis This study finds a condition and solves a problem in the fie ld of optics. But the methods are partly based on computer science and information technique. Thereby, some background knowledge needs to be introduced b riefly. First of all, an appropriate model should be chosen to represent images. In this study, we adopt a classic model wid ely used in the field of Di gital Image Processing [15]. An im age is divi ded into several uniform grids, then each grid is treated as a pixel, and its light intensity is called a pix el value. As a result, the image is represented as a matrix. The matrix (digital image signal) i s an approx imation of ph y sical im age in given samplin g ra te and quantization a ccuracy. The structure information of samples, which is what people concern, is carried in the corresponding digital image signals. In this case, our task is not to localize luminous points accurately, but to figure out the light i ntensity (pixel value) of each grid instead. The core of this study is based on the following phenomenon: information can be carried in the same signal in different ways. There is a common opinion in the field of Di gital Image Pro cessing: 3 / 24 a sample’s profil e inform ation, which changes slowly in space, corresponds to the low frequency part of its image’s Fourier spectrum; while its detail information, which changes fast in space, corresponds to the high frequency part. This is true in usual imag ing condition because each pixel value corresponds directly to a grid in image area. Thereb y, the spatial structure info rmation is carried directly in pixel values. However, the situation might be di fferent if the information is carried indirectl y . Strictly speaking, bo th high frequency and low frequenc y components are concepts attached to signals rather than information. The y do demonstrate f ast -changing or slow- changing forms in spac e domain. But they might n ot necessarily correspond to the profile or details of a sample if the information is not carried directly. Here are some simplified examples about information and its ca rriers. Example 1: if two physical points are used to carry in formation, their amount could represent the integer- v alue “2”, or their distanc e could represent a rea l-value such a s 123.822253. In this exa mple, the information carriers are ph y sical objects. In many other cases, observed si gnals are used to carry information. Example 2: in a Single-Molecule-Localization microscope , the observed image of individual molecules is blurred, and the pixel values do not show the molecules’ detailed structure dire ctly. But what people concern are the mol ecules’ location s and lig ht intensities carried by the pixels indirectly. Such information can be e xtracted, with methods such as data fitting, when the microscope’s Point Spread Function (PSF) is known. In both of the ex amples, prior knowledge plays a key role, a nd determines how th e information is carried in the signals. In example 1, it tells whether the information is ca rried in the amount or the distance of the two points. In e xample 2, it provides the template required for data fitting, i.e., the PSF. B esides the above e xamples, there are more researches relevant to how information is carried in signals in indirect or implicit ways. We find that in a certa in condition, observe d images alwa ys carr y the full information of a sample’s structure, no matter they are sharp or diffrac tion-blur red. Therefore, the c ondition is termed “resolvable condition” here, a nd it has two aspects. The first aspect is named isolated lighting (or separated lighting ). I t means the Region of Interest (ROI) in the sample’s image is only affected by its own structure and lighting, and is independent of the rest of the sample and the whole surroundin g. For example, only one small area of the sample is li ghted, or onl y one molecule is turned on, whi le the rest part and the surroundi ng are either totally dark or have no light collected b y the micr oscope. I n practice, an R OI is treated to fulfill isolated li ghting as long as the effect of the rest part and the surrounding is i gnorable. For example, all the other light sources are far enough away from the ROI , similar to what happens in some super-resolution techniques but may n eed to be stricter. Such a condition is not difficult to implement with e xisting techniques. B ut it actually provides very strong prior knowledge because it determines infinitely many pixel values (i.e., zeros) outsi de the ROI. The convolution in a light microscope is usually e xpressed by the f ollowing equation:          4 / 24 But the observed pa rt of the sample, i.e., the ROI onl y has limited size in real applications. Thereby it is also a ffected by any extra light from other parts or the whole surrounding, espe ciall y the structures around the ROI. In this case, the above equation should be modified as follows:             Where, “ extra ” means the extra li ght from other p arts or the whole surrounding. In usual case, it is an unknown signal, and its values mig ht be large a nd affect the re sult significantl y . It tends to be larger when zooming fac tor is larger and points’ images overlap more s everely. Actuall y , the extra li ght could even be much greater than the true signal itself when z ooming factor is ver y large. For example, Fig. 1a and 1b show samples in normal condition and isolated lighting conditions, respectively. The dashed-line rectangles in all figures indicate the ROI, which is    pixels. It simulates a phy sical region of    nm. Thereby, the Airy-disk is about    pixels, i.e.,    nm under this zooming factor. Then, Fig. 1c and 1d is the convolution results of 1a and 1b , respective l y. The ROI’s light intensities in 1c are overlapped b y th e images of outside structures. But that in 1d is affe cted by nothin g else outside the ROI because the surroundings are all dark . As a r esult, the R OI’s light int ensities in 1c are much greater than that in 1d . Actuall y , quantitative anal ysis shows that the former is more th an 8.87 times greater than the latter. That means the extra light (unknown) is more than 7.87 times greater than the true signal, and submerge s it overw helmingly. There b y , the ideal image c annot be fig ured out from the equation even when there is no noise at all. But i n the condition of isol ated lighting , e.g., in Fig. 1d , the extra light is zero. Therefore, a significant barrier on this approach is entirely eliminated. The second aspect is named positive effective P SF , which means that all the values of the effective PSF are positive (i.e., grea ter than z eros). Where, effective PS F means the part of PS F which affects the convolu tion results in the ROI. This aspect might be fulfilled in various situations. For example, the e ffective PSF values are, of course, positive if the PSF is to tally positive; thi s is a little stricter than the s ituation in usual applications [16] . Or, the PSF may have non -positive values at its “dark rings”, but only the central part of the PSF (i.e., the Airy disk) affects the convolution results in the ROI when the RO I is smaller than the diffraction- limit . In this case, the other part of the PSF would only affect the convolution results outside the RO I . Thereb y , the effective PSF is the central part, whose values are usually all positive, for normal light microscopes . In practice, approximate solut ions might be fig ured out sometimes even if the conditions are not fulfilled strictly, but the effective ness would be uncertain. The pixels of sharp image s carry the full information directl y , including both profile and details . The blurred images carry not only the profile inf ormation directly, but also the full information indirectly in the “resolvable condition”. Such a situation of “ one carrier, two types of info rmation” is somewhat similar to the above example 1. Different ways of c arr y lead to different methods for extraction. F ull information can be observed direc tly in sh arp image s. (Strictly speaking, microscopic images cannot be full y “sharp” b ecause the y are always the results of convolution due 5 / 24 to diffraction. F ortunately, errors are ignorable when points are distant enough from one another). But more steps might be required to extract full information from blurred images, e.g., solvi ng a system of equations. This stud y is based on the aforementioned image model, no matter for space domain or frequency domain. Therefore, the task of information ex traction is translated into the calculation of unknown pixel values, i.e., matrix e lements. The rough locations of unknown pixels in the imag es should be estimated first, and this could be done using existing techniques such a s Single-Molecule-Localization. Fig. 1 . The effect of ex tra lights on RO I’s convolution results. (a) A sample in normal condition. (b) A sample in isol ated lighting condition. (c) The convolution result in normal condition. (d) The convolution result in isolated lighting condition . The three dashed-line rectangles indicate the ROIs. The above phenomenon can also be ex plained in another wa y. Assume that a signal (e.g., an image) undertakes a certain processing (e.g., convolution). The ideal (orig inal) signal can be recovered from the r esult if the processing is reversible. However, a c onventional light microscope b a c d 6 / 24 filters out the high frequency part of an y image, thereby the ideal image cannot b e recovere d reversibly. As mentioned before, the low frequency part of a signal represents its profile, and the high frequency part represents its details. Therefore, de tails (smaller than the diff raction-limit) are lost after ima ged b y the mi croscope. But the processing ( convolution caused b y th e microscope) is reversible in the “resolvable condition”. As a result, the details can be recovered f rom the blurred image or the low frequ ency part. In the following sections, two recovering methods will be described, for space domain and frequenc y domain respec tivel y. Noises are neglected h ere and considered separately. 2.2. Method for spatial dom ain The effect of diffrac tion on imag ing is usually modeled as the convolution of PSF with ideal images. I t diffuses the light intensity of each pixel to other pixels, and ther eby lowers the diversity of pixel values, i.e., blurs the images. F or convenience, we first explain the method on 1D sig nals. Analogous to the aforementioned image model, a 1D range is also divide d into several uniform segments, and e ach segment is represented b y a value. L et’s tak e a simple signal as an example, as shown b y Fig. 2 . It has two values which are greater than z ero, while al l the other values are zeros. This is an analogy to the situation that there are onl y two lumino us point. Fig. 2 . The 1D situation of the sp atial domain me thod. (a) Before convolu tion (the ideal signal). (b) During convolution. (c) After convolution (the observed signal). Where, Fig. 2a shows th e 1D si gnal before convolut ion, which named “ideal signal” here. The two values in dash line need to be figured out. Fig. 2b shows the situation during convolution, where the convolution kernel is a lready kno wn. It is equivalent to the PSF of a 2D imaging s y stem, and ca lled Impulse Re sponse Function (I RF) here. Fig. 2c shows the resulting signal after convolution. I t is also known alread y , and named “observed signal” here. The result of convolution, i.e., the observed signal looks like the IRF. It i s much smoother than the ideal signal, which comprises two impulses. For image si gna ls, being smoother usually means being more blur red, and harder to identify their details. a b c 7 / 24 However, detail information can be recovered from the observed signal, in the “r esolvable condition” . For the 1D situation in Fig. 2 , the ROI covers th e two non -zero values (just like two separated pixels in an im age). The condition means the observed signal’s values in the ROI are only relevant to the ideal signal’s values in the ROI , and the IRF ’s values. No other values of the ideal signal can affect the convolution result because they are all z eros. In this case, the observed signal is the convolution of the ideal signal with the IRF. Therefore, the t wo unknown values in Fig. 2a ca n b e figured out from the known I RF and observed signal. (Otherwise, if the “re solvable condition” is not fulfilled, there might be unknown luminous points at unknown locations outside the RO I . Their images, which ex tend infinitel y , would overlap the ROI . Thereb y , the aforementioned relationship would be incorrect). Denote: 1. T he ideal signal’s unknown values are   and   at the left and right, respectively; 2. The amplitude, i.e., the central value of the IRF is  ; 3. The I RF has value   at the location o f   when its center is at the locati on of   . Therefore, the value      is shown by the left dark diamond in Fig. 2b . 4. The I RF has value   at the location of   when its center is at the location of   . Therefore, the value      is shown by the right dark diamond in Fig. 2b . 5. The observed signal has values   and   at the location of   and   , respectively. Since the observed signal is the convolution of the idea l signal with the I RF, we get the following system of equations: 󰇥                           (1) The solution of the above system of equations is: 󰇱                                         (2) Wh en    and the IRF is symmetrical, i.e.,        , formula (2) becomes: 󰇱                     (3) For e xample, suppose that the ideal signa l is 󰇛     󰇜 , where the suspension points mean infinitely man y zeros; the I RF is 󰇛        󰇜 , where th e suspensio n points mean arbitrar y values (they do not a ffect the result in this case) ; th erefore,    ,         ; then assume that the observed signal is 󰇛       󰇜 , where 4.5934 and 4.5977 are the valu es of   and   respectively, and suspension points mean the other values. Substitute these values into for mula (2) and we get:     and      . In other words, the ideal signal is recovered from the observed signa l and the IRF. 8 / 24 It can be seen that onl y part of the observed signal and the I RF is used. I n fact, it is not necessary to use   and   which have the corresponding location to   and   . The method also works if values at other locations are chosen from the observed signal. In practice, it ma y be he lpful for relieving the effect of observation errors if more va lues are used to build an o verdetermined s y stem of equations. Now we will extend the procedure to 2D sig nals, e.g., the situation of 2D imaging, as shown b y Fig. 3 . Where, Fig. 3a shows the image in ideal condit ions, i.e., without the effect of diffraction. Such an image is named “ideal image” in this case. F ig. 3b shows the image observed b y the microscope , which is named “observed image”. I t is the convolution result of the idea l image with th e PSF, due to diffraction. Fig. 3. The 2 D situation of the spatial domain method. (a) Before convolution (the ideal image). (b) After convolution (the observe d image). The two dashed-line rectangle s indicate the ROI. It can be seen that all the pixels in the ideal im ag e equal zeros except in a rectangular ROI. Therefore, the condition of isolated lighting is fulfilled. The P SF’s Fourier spectrum is an ideal low pa ss filter, therefore the PSF extends infinitely in space domain. B ut the PSF’s e nergy or light intensity is mainly concentrated in the central area. The observed image is extremely blurred, and looks similar to the PSF. It is ve ry difficult to see any details in the observed image, especiall y i n the R OI , which i s indicated b y a dashed-line rectangle. I n fact, the condition of isolated li ghting does not restrict the ROI ’s shape. Eve n an ROI with man y disconnected and irregular areas is acceptable, as lon g as its rough location could b e estimated in the observed im age. However, one easy way is to find a rectangle to cov er all the areas, and then treat the rectangle as the ROI. I n order to decrease the complexit y of calculation, the rec tangle should be as small as possible. Similar to 1D situation, the observe d image ’s pixels in the RO I are onl y relevant to the ideal image’s pix els in the ROI and the pixels (values) of P SF. The other pixels in the ideal image do b a 9 / 24 not affect the result of convolut ion because the y are all zeros. In this case, the observed image is the convolution of the ideal image with the PSF. Therefore, the unknown pixels in the ideal image can be figured out from the known PSF and observed image. Let’s: 1. T reat the ideal image’s RO I as an ima ge named  󰇛    󰇜 ,        and        ; where  and  are the amount of the ROI ’s row and column.  and  start from 󰇛   󰇜 at the ROI ’s most left -top pixel. 2. Denote the PSF as image  󰇛   󰇜 , and set a coordinate with its origin at the PSF’s center. Thereby, the pix el value is  󰇛   󰇜 at the PSF’s center, and both  and  belong in 󰇟    󰇠 ; 3. T reat the observed image’s ROI as an image na med  󰇛   󰇜 , where        and        . Let’s take a rotationally symmetrical PSF as an example, and the other PSF could be handled similarly. During convolution , let the PSF overlap the ideal im age, and ali gn the PSF’s center with each pixel in the ideal image’s RO I each time. Then, multipl y each pixel in the ideal image’s ROI with it s corresponding PSF pixel, and accumulate all the results. The acc umulative value is the observed image’s pixel value in the corresponding location. Th e above procedure could b e implemented concisely with a program. In mathematics, this could be expressed with a s y stem of linear equations as follows:    (4) Where:         󰇛  󰇜   󰇛     󰇜   󰇛    󰇜   󰇛       󰇜         󰇛     󰇜   󰇛  󰇜   󰇛       󰇜   󰇛    󰇜         󰇛    󰇜   󰇛       󰇜   󰇛  󰇜   󰇛     󰇜         󰇛       󰇜   󰇛    󰇜   󰇛     󰇜   󰇛   󰇜      This is a matrix with a size of 󰇛    󰇜  󰇛    󰇜 . Then:         󰇛  󰇜   󰇛   󰇜   󰇛    󰇜   󰇛    󰇜      This is a matrix (vector) with a size of 󰇛    󰇜   , and it is actually a sequ ence of a ll the pixels in the ideal image’s ROI, arranged row by row from top to bot tom. Then: 10 / 24         󰇛  󰇜   󰇛   󰇜   󰇛    󰇜   󰇛    󰇜      This is also a matrix (vector) with a size of 󰇛    󰇜   , and it is actuall y a sequence of all the pixels in the observed image’s ROI , arran ge d row b y row from top to bottom. The above  is determined by the PSF, and  is determined b y the observed image. In other words, both  and  are a lready known, and  is actually the rearrangement of the ideal image’s unknown pixels. In theory, if we substitute the ideal im age’s vector  into formula (4 ), the resulting  sh ould equals  . Accordingly, the ide al image can be get by sol ving formula (4). In the above procedure, only the pixels in the observed image’s RO I are a dopted. Actually, if the other pixels of the observed im age a re also used, an overdetermined system could be buil d including more eq uations. In practice, that ma y be helpful for improving the method’s capability of noise resistance. In this case, the unknowns are still the idea l image’s pix els in the ROI because all the other pixels are known to be zeros, in the condition of isolated lighting. There are many classic or cutting-edge methods can be adopted for solving the above system of equations. I ts solvability can b e explain as follows. On the one hand, th e s ystem of equations should obviousl y have at least one solution, i.e., the idea l im age itself. On the other hand, the “resolvable condition” h as a sec ond aspe ct, i.e., effective PS F is totally positive. Th erefor e, the elements of matrix  are all positive. In addition, the elem ents of vector  , i.e., t he ideal image’s pixels are all non-negative also, thereby    is true only when    . In oth er words, the system of homogeneous linear equations    onl y h as the zero solution. Therefore, according to the propert y of sy stem of linear equations, the corresponding system of nonhomogeneous line ar equations    has only one solution (i.e., the ideal image) [17]. In th e above procedure, the ideal image is figured out by buildin g and solving a system of equations. This seems to be unrea sonable in frequency domain because the hig h frequency part of the image has been filter ed out by the microscope. However, we find that the full information of a sample’s spatial structure can be recovered even with onl y a small amount of frequenc y el ements in the “resolvable condition” . 2.3. Method for frequency dom ain According to Fourier Optics, convolution caused by diffraction is equivalent to filterin g ideal images with an ideal low pass filter. Assume that the low pass filter’ s amplitude is 1, without loss of generality. First, let ’s describe the method on a simple 1D signal, as shown by Fig. 4 . 11 / 24 Fig. 4. The 1D situation of the frequency domain method. (a) The ide al signal including only two non-zero values. (b) The corresponding spectrum, where onl y the most left two values are preserved. Where, Fig. 4a shows an ideal signal. I t has only t wo unknown values, while all the other values are zeros. Therefore, the c ondition of isolated lighting is fulfilled. Fig. 4b shows the ideal signal’s Fourier sp ectrum. A fter low pass filtering, all the high frequency components bec ome zeros . Therefore, the known spectrum only includes low frequency components, and is named “observed spectrum” her e. For ex ample, the most left tw o values in the spectrum are preserved, and treated as the observed spectrum. In 1D case, t he low pass filter is equivalent to the IRF’s Fourier spectrum, which is usuall y c alled S y stem Transfer Function (STF). I n this case, th ere is a mathematical relationship between the ideal signal and the observed spectrum. Assume that: 1. The length of the ideal signal is  , and    ; 2. T he idea l signal’s unknown values are   and   respectively, where both  and  are integers within 󰇟     󰇠 ; 3. Arbitrarily choose two values   and   from the observe d spectrum, where both  and  are integers. The formula of 1D discrete Fourier Tra nsform is as follows [18]:                 (5) Sin ce all the ideal signal’s values are zeros except   and   , the above formula becomes:                    (6) Let  equals  and  respectively, then substitute them into formula (6), and we get:                                      (7) So lve the above system of linear equations, and we get:                󰇛    󰇜     󰇛    󰇜   (8) and: b a 12 / 24              (9) For example, assume that the ideal signal is 󰇛          󰇜 , and thereb y the corresponding Fourier spectrum is 󰇛            󰇜 . I n the latter , the complex values   and         are the chosen components of the observe d spectrum, and the suspension points represent the other components. Therefore,    ，    ，    ，    ，    ，      ，            . Please not e that       are all coordinate indices of 1D sig nals, which start from 0. Substitute the above values into formulas (8) and (9), and we get     and      . In other words, the i deal signal is recovered from the observed (low frequency) spectrum. It can be seen that onl y two freque ncy c omponents of the observed spe ctrum are used. Actually, they could be chosen from the low freque nc y part, or the high fre quenc y part, or even arbitrary part. This method works as long as a s ystem of equations is buil t and a unique solution is got. In practice, it may be he lpful for relieving the effect of e rrors if more frequenc y components are used to build an overdetermined system of equations. Now we will extend the procedure to 2D sig nals, e.g., the situation of 2D im aging. I n this case, the STF is called Optic al Transfer Function (OTF) which is equivale nt to the PS F’s Fourier spectrum, as shown by Fig. 5 . Fig. 5. Th e 2D situation of the frequenc y domain method. (a) Th e ideal im age including a small ROI. (b) The corresponding freque nc y spec trum, where the low frequency part is encircled. Where, Fig. 5a shows an ideal 2D signal without being filtered, which is named “ideal image” here. F ig. 5b shows the ideal image’s spectrum of 2D discrete Fourier transform. After b ein g filtered by an ideal low pass filter, the spectrum’s components outside the dotted circle are all b a 13 / 24 removed. Thereby, the known spectrum includes components inside the circle onl y , and is named “observed spe ctrum”. P lease note that the spectrum is shown conventionally in Fig. 5b , i.e., the low frequency components are at the ce nter. But it is still handled in orig inal format in the following procedure, i.e., the high frequency components are at the center. In this case, there is a mathematical relationship between the ideal image and the observed spectrum. Assume that: 1. The idea l image is  󰇛   󰇜 ;        ，        , where  and  are the amount of the image’s row and column , respectively ;  and  start from 󰇛   󰇜 at the image’s most left -top pixel; 2. T he size of the ideal image’s ROI is    , and the row number and the column number of the ROI’s left -top pixel are  and  , respectively; 3. T he ideal image’s full spectrum is  󰇛   󰇜 , where        ，         ; 4. Choose a rectangular area from the observed sp ectrum, with a size of    ; the row number and the column n umber of it s left-top pixel are  and  , respectivel y ; for example,      . The formula of 2D discrete Fourier transform is as follows [15]:  󰇛   󰇜        󰇛   󰇜     󰇡      󰇢     (10) Since all the ideal imag e's pixels are zeros ex cept those within the R OI , the above formula becomes:  󰇛   󰇜        󰇛   󰇜     󰇡      󰇢       (11) Substitute the chosen components of the observed spectrum into formula (11), and we get a system of equations    , where :                󰇡      󰇢         󰇛  󰇜        󰇛  󰇜            󰇛   󰇜    󰇛  󰇜                   󰇛  󰇜           󰇛  󰇜  󰇛  󰇜        󰇛   󰇜     󰇛  󰇜        󰇛  󰇜    󰇛  󰇜  󰇛  󰇜               󰇛   󰇜            󰇛   󰇜   󰇛  󰇜         󰇛  󰇜  󰇛   󰇜           󰇛   󰇜  󰇛   󰇜   󰇛  󰇜                󰇛   󰇜    󰇛  󰇜         󰇛   󰇜   󰇛  󰇜  󰇛  󰇜        󰇛   󰇜  󰇛   󰇜    󰇛   󰇜        󰇛   󰇜  󰇛   󰇜   󰇛  󰇜  󰇛  󰇜          This is a matrix with a size of 󰇛    󰇜  󰇛    󰇜 . Then:         󰇛   󰇜   󰇛        󰇜   󰇛       󰇜   󰇛           󰇜      14 / 24 This is a matrix (vector) with a siz e of 󰇛    󰇜   , and it is a sequence of all the pixels in the ideal image’s ROI, arranged row by row from top to bottom. Then:         󰇛    󰇜   󰇛       󰇜   󰇛       󰇜   󰇛           󰇜      This is also a matrix (vector) with a siz e of 󰇛    󰇜   , and it is a sequence of the chosen components of the observed spectrum, arranged row by row from top to bott om. Similar to the method for spatial domain, vector  can be got by solving    , and then the ideal image can be got by the rearrangement of  . Although the rectangular area are chosen from the observed spectrum, other shape s or even randomly chosen components are also a llowed in this method. This method works as long as a s y stem o f equations is buil t and a unique solution is got. In practice, it may be helpful for relieving the effect of errors if more freq uency components are used to build an overdetermined system of equations. It can be seen that lar ger ROI could be r ecovered if more spectrum components are p reserved after filtering. I n a special (extreme) case, all the spectrum components are known. Thereby, the ideal image could be got directly by inverse filtering or inverse Fourier transform. Actuall y , it is a classic way recovering a full image from its full Fourier spectrum. This study find s more possibility: recovering part of an im age (with full details) from part of its Fourier spectrum (e. g., including only low frequenc y ). That means even low frequency components carr y full details of a s ample's spatial structure (in an RO I), and seems inconsistent with traditional opinions. After excluding several explanations, we believe that the reason is the "integrity of sp ectrum", i.e., di fferent frequency components are tightly relevant in the “resolvable condition” . Let's take 2D case as an example. As can be seen from the formula of 2 D discrete Fourier trans form, the spectrum is actually the accumulation of the products of each pixel  󰇛   󰇜 with its corresponding basis function in frequency domain. Each product is as follows:     󰇛   󰇜     󰇡      󰇢 This is a function including all fre quenc y components, and its amplitude is affected by th e corresponding pixel valu e  󰇛   󰇜 . W hen the pixel value varies, the function’s value s change accordingly at any frequency with the same p ercentage. In other words, each pixel value is c arried on the amplitude of its corresponding function, or the function’s value carries its cor responding pixel value at each frequency (from the lowest to the highe st frequency). Taking an extreme situation as an example, there is only one fun ction (product) in the image’ s spectrum when th ere is only one pixel in the image ’s ROI . In other words, the spectrum is the product of the only pixel with its corresponding b asis function. When  ,  and the pixel’s location 󰇛   󰇜 is known, the 15 / 24 basis function       󰇡      󰇢 is also known. Therefore, pick the observed spectrum’s value at an arbitrarily selected freque nc y, and divided it b y the basis function’s value at the same frequency, then the resul t is the unknown pix el value  󰇛   󰇜 . Actually, the s elected frequency could even be zero frequenc y, i.e., DC component in this case. This situation is similar to that when an individual molecule’ s light int ensity is extracted using Singl e-Molecule-Localization techniques. When there are more unknown pixels, they ca n be figured out by building and solving a s ystem of equations, as shown b y the aforementioned procedure . From this point of view, thi s technique could be treated as the extension of existing techniques such as Single -Molecule- Localization, and it furthe r “ split” a single point into    ,    or more points, i.e., the ROI. It can be seen th at the full frequency spectrum is relativel y redundant when the ROI is sm aller than the ideal image. In this case, there is no need to recover the full sp ectrum, as what popular deconvolution techniques do, for the recovery of the ideal image. A blurred image without high frequency components looks meaningless and le ss informative, but it actuall y contains the full information of the ideal image, in the “resolvable condition” . I n oth er words, after diff erent ideal images are filte red, the r esulting bl urred im ages all seem similar and undistinguishable. But the y are actuall y different from one another , as can be seen from formula (10). From the information theory ’s point of view, that means they actua lly carr y different information, i.e., the y are distinguishable [19]. 3. Results and discu ssion Fig. 6. The PSF and its Fourier spectrum. (a) The experimental PS F. (b) T he PSF’s Fourier spectrum which is an ideal low pass filter. Frequency components are all removed if they are more than 6 pixels away from the center, i.e., zero frequency. a b 16 / 24 Simulation experiments are performed for spatial domain and frequency domain respectivel y . Each ideal image (expected imag e) tested is of size    , and has an RO I in it. Then, 19 different siz es of RO I, i.e.,    ,    ,  ,    pixels, are tested for both spatial domain and frequency domain respectively. Furthermore, 20 random tests are performed for each size. In order to cover various possibilities, t he pixel value s in the R OI are randomly generated in each test. There fore,      differe nt ideal image s are tested f or spatial domain and frequency domain, respectively. In theor y , the Airy-disk-shaped PSF extends infinitel y because the microscope is an ideal low pass filter. But according to the analysis in METHODS, the convolutio n result in the ROI is affected onl y by th e PS F’s cent ral area when the PSF is lar ge enough. Therefore, the PSF o f siz e     is adopted in this experiment as shown by Fig. 6a . The PS F’s Fourier spectrum is shown by Fig. 6b . 3.1. Results for spatial domain Fig. 7. The Averaged Errors of the spatial domain method. Where, lateral axis shows the R OI ’s size, from    to    pixels. Vertical axis shows the Av eraged Error. The error b ars show the standard deviations of the Average Errors. The value 1.00E+05 m eans      , and the other values are similar. Now we will describe th e ex periment for spatial domain. After convolution, all the observed images look blurred, just li ke Fig. 1d , but they actually carr y information different from one another. Fi gure out the ideal images’ unknown pixels using our method for spatial domain, and the 17 / 24 Averaged Errors (AEs) of the results are shown by Fig. 7 . There is an Averaged Error for each size, which means the ave rage of all the testing errors in the 20 tests. The testing error is defined as     󰆒  󰇛    󰇜  , which represents the mean square error between the recovered pixels and their corresponding pixe ls in the ide al image’s R OI . In the formula,  represents the v ector rearranged from recovered pixels,  󰆒 represents the vector rearranged from the pixels in the ideal image’s ROI , and    is the count of pixels in the R OI. All the Averaged Errors are also shown in Table 1 , as follows. Table 1. The Averaged Error s (AE s) of the spatial domain method. The value 3.23 E-08 means      , and the others are similar. It c an be s een from Table 1 that the averaged errors are very tiny for size s    and    . Given the upper limit of pixel value is 256, the recovered results can be treated as the same as the corresponding ideal images. That verifies the effectiveness of the method: the spatial resolution is increased by 3 times in ea ch dimension if each individual pixel is re solved into    pix els with full details. For sizes    to    , the averaged err ors are larger, and it is hard to judge whether the method still works in these cases. Therefore, w e calculate the Avera ged Difference (ADs). There is an Averaged Difference for each siz e, which means the average of all the differe nce values in the 20 tests. Where, the difference value is de fined as      󰇛    󰇜  , which reflects how well the vector  fulfills the formula    . In thi s experiment,  represents the vector rea rranged from observed pix els in the RO I . By checking the difference value, we can see how we ll the ideal image fulfill the corresponding s ystem of equations. For each size of ROI , the difference values are averaged for all the 20 random tests. The resulting Average d Differences are shown in Table 2 . In addition, the st andard deviations of difference v alues are less than    for all the above sizes. According to these results, the system of equations in this method can model the imaging procedure a ccurately ROI size  ×   ×   ×   ×  AE 3.23E- 08 0.21 202.93 236.88 ROI size 6 × 6 7 × 7 8 × 8 9 × 9 10 × 10 AE 458.63 1778.16 1634.61 2649.79 1652.73 ROI size 11 × 11 12 × 12 13 × 13 14 × 14 15 × 15 AE 1909.78 2252.56 3667.90 3863.32 6898.65 ROI size 16 × 16 17 × 17 18 × 18 19 × 19 20 × 20 AE 3835.98 12051.00 7861.40 49128.85 21946.59 18 / 24 enough. That suggests that the large errors in Ta ble 1 are not caused b y the method’s p rinciple. More accurate results could be got if more effective approache s are used to solve the s y stem of equations. The refore, the method’s effectiveness is also proved indirectly for size    to    . Table 2. The Averaged Differences (ADs) of the spatial domain method. 3.2. Results for frequency domain Fig. 8. The Averaged Errors of the frequenc y domain method. Similar to the experiment for spatial domain, the lateral ax is shows the RO I’s siz e, and the vertical axis shows the Averaged Errors. The error bars also show the standard deviations of the Average Errors. ROI size 2 × 2 3 × 3 4 × 4 5 × 5 AD 2.43E- 17 2.48E- 17 3.98E- 17 4.86E- 17 ROI size 6 × 6 7 × 7 8 × 8 9 × 9 10 × 10 AD 7.30E- 17 1.03E- 16 1.09E- 16 1.60E- 16 2.00E- 16 ROI size 11 × 11 12 × 12 13 × 13 14 × 14 15 × 15 AD 2.12E- 16 2.44E- 16 3.42E- 16 4.00E- 16 4.12E- 16 ROI size 16 × 16 17 × 17 18 × 18 19 × 19 20 × 20 AD 4.26E- 16 4.58E- 16 5.91E- 16 6.99E- 16 7.62E- 16 19 / 24 Then we will des cribe the experiment for frequenc y domain. In this experiment, the observed image’s frequenc y spectrum is zero except its low frequenc y part. Fi gure out the ideal imag e’s unknown pixels using our method for fr equency domain. The resulting Averaged Errors (A Es) a re shown in Fig. 8 , and Table 3 . Table 3. The Averaged Errors (AEs) of the frequency domain method. As can be seen from Table 3 , the Ave raged Errors are very tiny for size s    to    . Therefore, the recove red re sults c an be treated as the same as the corresponding ideal images. That verifies the effectiveness of the method: the spatial resolution is incr eased by 4 ti mes in each dimension if each pixel is resolved into    pixels with full details. The errors for size    in the spatial method are larger than those here. But both the two methods are based on the same principle. This suggests that the larger errors are not caused by the principle. Table 4. The Averaged Differences (ADs) of the frequency domain method. Then, the Averaged Differences (ADs) are also calculated, as shown by Table 4 . In addition, the standard deviations of difference values are less than    for all the siz es. According to ROI size 2 × 2 3 × 3 4 × 4 5 × 5 AE 1.21E- 09 1.63E- 04 0.59 118.56 ROI size 6 × 6 7 × 7 8 × 8 9 × 9 10 × 10 AE 1432.21 2977.23 7357.42 3455.11 28796.84 ROI size 11 × 11 12 × 12 13 × 13 14 × 14 15 × 15 AE 14055.47 5611.23 1518.55 5025.37 1011.36 ROI size 16 × 16 17 × 17 18 × 18 19 × 19 20 × 20 AE 1632.88 3181.26 12717.13 641.46 2037.52 ROI size 2 × 2 3 × 3 4 × 4 5 × 5 AD 5.86E- 14 1.72E- 13 2.23E- 13 2.86E- 13 ROI size 6 × 6 7 × 7 8 × 8 9 × 9 10 × 10 AD 5.11E- 13 5.63E- 13 9.39E- 13 1.13E- 12 1.39E- 12 ROI size 11 × 11 12 × 12 13 × 13 14 × 14 15 × 15 AD 1.58E- 12 2.20E- 12 2.51E- 12 2.79E- 12 3.18E- 12 ROI size 16 × 16 17 × 17 18 × 18 19 × 19 20 × 20 AD 3.46E- 12 3.77E- 12 4.03E- 12 4.15E- 12 4.37E- 12 20 / 24 these results, the s y stem of equations in this method can model the imaging procedure accuratel y enough. More accurate results could be g ot if more effective approaches are used to solve the system of equations. The refore, the method’s effectiveness is also proved indirectly for siz es    to    . Furthermore, the methods are also verif ied indirectly for size    to    . Given the li mitation of computational resource and time, only one random test is performed for e ach size. All the results demonstrate that the methods can model the imaging procedure pre cisely b ecause the Averaged Difference is smaller than    in eac h test. In practice, the    ROI can be a single luminous point such as a fluorescent molecule. Fig. 9 shows the zoomed view of an experiment result. The    pixels at the center o f Fig. 9a or Fig. 9c form the ROI, whic h is shown as a square at the center of Fig. 9b . After convolution, the observed im age actuall y extends infinitely, a nd it is very blurred. Thereby, the pix els in Fig. 9b look almost all the same, but the y are different slightly. Existing techniques could be used to estimate the RO I’s location, which is indi cated by the asterisk in Fig. 9b . The recovered image is almost the same as the ideal image, and includes both profile and detail information. Fig. 9. The zoomed vie w of an experiment result. (a) I deal image (before convolution). (b) Observed image (after convolut ion). (c) Recovered image. In all the three images, dot -line grids represent the borders of pixels. The asterisk in (b) indicates the estimated location of the ROI. An experiment is also perf ormed on larger ima ges, and the result of an example is shown by Fig. 10 . Where, Fig. 10a is the ideal im age , which sim ulates the sample’s physical structure including various objects. Although it is a sy nthes ized image (    00 pixels), it is enough to test the proposed technique in princ iple. Fig. 10b is the direct c onvolution of the ideal imag e with the PSF, and it simulates the ima ge from a conventional microscope. F ig. 10c is th e result recovered with the proposed technique. For this si mulated microscope, assume that its resolution is 5 nm per pix el. The sample (ideal im age) is sca nned with a luminous poi nt of    , just like what happens in STED. Thereb y , the RO I is    pixels at each time, and the id eal image (    pix els) is scanned for totally    times. The Air y- disk’ radius is approximately betw een 2 00 to 300 nm, thereb y it is betwee n 40 and 60 pixels. The conventional 21 / 24 result ( Fig. 10b) is very blurred (even hard to se e meanin gful profile ) because the Airy-disk is much larger than any details. But the recovered image is sharp, with full details, and the averaged error is ignorable (about 0.0045 of the averaged pixel value of the ideal image). As the resolution of a conventional microscope is about 200nm, the improvement of resolution is about 200/5 = 40 times in this experiment. In principle, the whole i mage could be tre ated as an ROI and recovered in one pass, as long as the “resolvable condition” is fulfilled. But in practice, it would be difficult to solve the large system of equations (    00 unknowns in this example) acc urately enough. Fig. 10. An example of larger images. (a) Ideal image (sample structures). (b) Convolved directly (conventional microscope). (c) Recovere d with the pr oposed technique. The recovered image would have similar accuracy to the simulation experiment if the observed image and P SF are completel y accurate. But noises are almost inevitable in practice. A preliminary experiment shows that the recovered image is acceptable when PSNR (Peak S ignal- to -Noise Ratio ) is gre ater than a certain th reshold such as 250. I t can be seen that the technique is sensitive to noise, thereby further research on SNR improvement would be necessary . There are m any valuabl e methods in fields such as Noise Reduction which ma y be utilized. Besides, the proposed technique recovers detail informati on directl y f rom an image without high frequency components. But noises usually c ontain more high fr equency components. This diffe rence may a lso be helpful to suppress noises. 4. Conclusion Usually, an image’s low f requency part r epresents its prof ile, and high frequency part repres ents its details. A conventional light microscope filters out hig h frequency part, and thereby make s the details cannot be recovered. In other words, t he convolution caused by the microscope is irreversible. Bu t thi s st udy finds a “ resolvable condition” wh ich is relevant to the imaging condition of existing techniques. In this condition, the observed image (or its Fourier spe ctrum) has redundan cy . The reby, the details can be recovered from part of the observe d image, or part of its Fourier spectrum (e.g., low frequency c omponents). In other words, a sample’s structure 22 / 24 (including both profile and details) can be extract ed directly from a blurred image, even if its size is much smaller than the diffrac tion-limit and its inner points are imaged at the same time. Thereby , the convolution is reversible and the recovery is independent of high frequency in this condition. Then, a tec hnique is proposed base d on the above findings and image deconvolution. The latter is usually employed for the recovery o f degraded images, e.g., relieving the effect of defocused light. Usually, it c annot recover the results of ideal low pass filters. But some classic deconvolution approaches are modified to a super-resolution tech nique when used in the “ resolvable condition” . It can extract sharp images from diffraction - blurred images directly, and get full details of samples’ spatial structure without using high frequency components. Ther efore, it is termed “Deconvolutional Super -resolution (DeSu - re)”. P opular convolution methods such as wiener filtering, usuall y re quires images not to be cropped. But the proposed technique can work even without any observed image outside the ROI. In principle, it could be used for resolving the inner structures of luminous points even if the y are infinitely small; or dist inguishing multiple points at the same time, even if they overlap one another. Therefore, this technique could a chieve unlimited resolutions in theor y . When combined with existing techniques, it can “split” a single point into multiple smaller points and thus improve s resolution b y several ti mes. The sim ulation ex periments directly verif y resolution improvement to 200 %~ 300% (i.e., 2~3 times) for the spatial domain method, and 200%~400 % (i.e., 2~4 times) for th e frequenc y domain method. In addition, they also indirectly verify resolution improvement to 200% ~10,000% (i.e., 2~100 times) for both methods. One of the futur e directions is to achieve higher reso lution a nd verify the effectiveness in practice. But there are still practical difficulties, especially the strong effect of the observed image ’s and PSF ’s dist ortion (e.g ., noise) on the results. Therefore, it is also an import ant future direction to get p ractical results as close to simulation results as possible. With the development of imaging devices and the improvement of signal -noise ratio, the accuracy of the proposed technique would also improve ac cordingly. The proposed technique could be combined with other techniques such as conventional microscope, confocal microscope, ex isting super -resolution methods, and so on. By extracting more details direc tly from the data of these techniques, higher resolution s or efficiency could be achie ved. For example, further resolve the inner details of individual molecules, fluorescent probes or tiny light sources after localization them. Or, several adjacent points (molecules, etc.) could be imaged and resolved at the same time, using the proposed tec hnique. Acknowledgements We thank THU, PKU, RUC, CAS, NLC, H DL, B ISTU, HH and H L for their r esources, information and support. Refer ences 23 / 24 [1] Sigal, Y. M., Zhou, R. B. & Z huang, X. W. Visualizing and discovering cellular structures with super-resolution microscopy. Science 361 , 880-887, doi:10.1126/science.aau1044 (2018). [2] Hell, S. W. & Wichmann, J. Bre aking the diffraction resolution limit by sti mulated emission: stimulated-emission-depletion fluorescence microscop y . Opt. Lett. 19 , 780-782, doi:10.1364/ol.19.000780 (1994). [3] Hofmann, M., Eggeling, C., Jakobs, S. & Hell, S. W. Bre aking the diffraction barr ier in fluorescence microsc op y at low l ight intensities by using reversibly photoswitchable proteins. Proc. Natl. Acad. Sci. U. S. A. 102 , 17565-17569, doi:10.1073/pnas.0506010102 (2005). [4] Gustafsson, M. G. Surpassing the lateral resolution limit by a f actor of two using structured illumination microscopy . Journal of microscopy 198 , 82-87, doi:10.1046/j.1365- 2818.2000.00710.x (2000). [5] Gustafsson, M. G. L. Nonlinear structured-il lumination mi croscopy : Wide- field fluorescence imaging with theoretically unlimited resolution. Proc. Natl. Acad. Sci. U. S. A. 102 , 13081-13086, doi:10.1073/pnas.0406877102 (2005). [6] Betzig, E. et al. I maging intracellular fluorescent proteins at nanometer resolution. Science 313 , 1642-1645, doi:10.1126/science.1127344 (2006). [7] Rust, M. J ., Bates, M. & Z huang, X. W. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM). Nat. Methods 3 , 793-795, doi:10.1038/nmeth929 (2006). [8] Sharonov, A. & Hochstrasser, R. M. Wide-field subdiffraction imaging by accumulated binding of diffusing probes. Proc. Natl. Acad. Sci. U. S. A. 103 , 18911-18916, doi:10.1073/pnas.0609643104 (2006). [9] Balzarotti, F. et al. Na nometer resolution imaging and trackin g of fluorescent molecules with minimal photon fluxes. Science 355 , 606-612, doi:10.1126/science.aak9913 (2017). [10] Chen, F., Tillberg, P. W. & Bo y den, E. S. Expansion mi croscopy . Science 347 , 543, doi:10.1126/science.1260088 (2015). [11] Eggeling, C., Wil lig, K. I., Sahl, S. J . & Hell, S. W . Lens-based fluo rescence nanoscopy. Quarterly reviews of biophysics 48 , 178-243, doi:10.1017/s0033583514000146 (2015). [12] Huang, B., Babcock, H. & Zhuang, X. B reaking the diffra ction barrier: super-resolution imaging of cells. Cell 143 , 1047-1058, doi:10.1016/j.cell.2010.12.002 (2010). [13] Sahl, S. J., Hell, S. W. & Jakobs, S. Fluorescenc e nanoscop y in cell biolog y . Nature reviews. Molecular cell biology 18 , 685-701, doi:10.1038/nrm.2017.71 (2017) . [14] Eldar, Y. C. & Pohl, V. Recovering signals from lowpass data. Trans. Sig. Proc. 58 , 2636- 2646, doi:10.1109/tsp.2010.2041278 (2010). 24 / 24 [15] Gonzalez, R. C. & Woods, R. E. Digital Image Proce ssing ( 3rd Edition) . (PEARSON, 2014). [16] Snyder, D. L., T.J . S chulz, and J .A.O. Sullivan, Deblurring subje ct to nonnegativit y constraints . IEEE Transactions on Signal Processing , 1992. 40 (5): p. 1143-1150. [17] Lay, D. C., Lay, S. R. & McDonald, J. J. Linear Algebra and Its Applications (5th Edition ) . (PEARSON, 2015). [18] Oppenheim, A. V. & Schafer, R. W. Digital Signal Processing . (PEARSON, 2015). [19] Sheffield, E.Y., Extracting supe r-resolution de tails directly from a diffraction-blurred image or part of its frequency spectrum . PeerJ Preprints , 2019. 7 : p. e27591v2.