用光訓(xùn)練神經(jīng)網(wǎng)絡(luò)，清華成果最新登上了 Nature！

無(wú)法應(yīng)用反向傳播算法怎么辦？

他們提出了一種全前向模式（Fully Forward Mode，F(xiàn)FM）的訓(xùn)練方法，在物理光學(xué)系統(tǒng)中直接執(zhí)行訓(xùn)練過(guò)程，克服了傳統(tǒng)基于數(shù)字計(jì)算機(jī)模擬的限制。

簡(jiǎn)單點(diǎn)說(shuō)，以前需要對(duì)物理系統(tǒng)進(jìn)行詳細(xì)建模，然后在計(jì)算機(jī)上模擬這些模型來(lái)訓(xùn)練網(wǎng)絡(luò)。而 FFM 方法省去了建模過(guò)程，允許系統(tǒng)直接使用實(shí)驗(yàn)數(shù)據(jù)進(jìn)行學(xué)習(xí)和優(yōu)化。

這也意味著，訓(xùn)練不需要再?gòu)暮笙蚯皺z查每一層（反向傳播），而是可以直接從前向后更新網(wǎng)絡(luò)的參數(shù)。

打個(gè)比方，就像拼圖一樣，反向傳播需要先看到最終圖片（輸出），然后逆向一塊塊檢查復(fù)原；而 FFM 方法更像手中已有部分完成的拼圖，只需按照一些光原理（對(duì)稱(chēng)互易性）繼續(xù)填充，而無(wú)需回頭檢查之前的拼圖。

這樣下來(lái)，使用 FFM優(yōu)勢(shì)也很明顯：

一是減少了對(duì)數(shù)學(xué)模型的依賴(lài)，可以避免模型不準(zhǔn)確帶來(lái)的問(wèn)題；二是節(jié)省了時(shí)間（同時(shí)能耗更低），使用光學(xué)系統(tǒng)可以并行處理大量的數(shù)據(jù)和操作，消除反向傳播也減少了整個(gè)網(wǎng)絡(luò)中需要檢查和調(diào)整的步驟。

論文共同一作是來(lái)自清華的薛智威、周天貺，通訊作者是清華的方璐教授、戴瓊海院士。此外，清華電子系徐智昊、之江實(shí)驗(yàn)室虞紹良也參與了這項(xiàng)研究。

消除反向傳播

一句話(huà)概括 FFM 原理：

將光學(xué)系統(tǒng)映射為參數(shù)化的現(xiàn)場(chǎng)神經(jīng)網(wǎng)絡(luò)，通過(guò)測(cè)量輸出光場(chǎng)來(lái)計(jì)算梯度，并使用梯度下降算法更新參數(shù)。

簡(jiǎn)單說(shuō)就是讓光學(xué)系統(tǒng)自學(xué)，通過(guò)觀察自己如何處理光線（即測(cè)量輸出光場(chǎng)）來(lái)了解自己的表現(xiàn)，然后利用這些信息來(lái)逐步調(diào)整自己的設(shè)置（參數(shù)）。

下圖展示了 FFM 在光學(xué)系統(tǒng)中的運(yùn)行機(jī)制：

其中 a 為傳統(tǒng)設(shè)計(jì)方法的局限性；b 為光學(xué)系統(tǒng)的組成；c 為光學(xué)系統(tǒng)到神經(jīng)網(wǎng)絡(luò)的映射。

展開(kāi)來(lái)說(shuō)，一般的光學(xué)系統(tǒng)（b），包括自由空間透鏡光學(xué)和集成光子學(xué)，由調(diào)制區(qū)域（暗綠色）和傳播區(qū)域（淺綠色）組成。在這些區(qū)域中，調(diào)制區(qū)域的折射率是可調(diào)的，而傳播區(qū)域的折射率是固定的。

而這里的調(diào)制和傳播區(qū)域可以映射到神經(jīng)網(wǎng)絡(luò)中的權(quán)重和神經(jīng)元連接。

在神經(jīng)網(wǎng)絡(luò)中，這些可調(diào)整的部分就像是神經(jīng)元之間的連接點(diǎn)，可以改變它們的強(qiáng)度（權(quán)重）來(lái)學(xué)習(xí)。

利用空間對(duì)稱(chēng)互易性原理，數(shù)據(jù)和誤差計(jì)算可以共享相同的前向物理傳播過(guò)程和測(cè)量方法。

這有點(diǎn)像鏡子里的反射，系統(tǒng)中的每個(gè)部分都能以相同的方式響應(yīng)光的傳播和錯(cuò)誤反饋。這意味著無(wú)論光如何進(jìn)入系統(tǒng)，系統(tǒng)都能以一致的方式處理它，并根據(jù)結(jié)果來(lái)調(diào)整自己。

這樣，可以在現(xiàn)場(chǎng)直接計(jì)算梯度，用于更新設(shè)計(jì)區(qū)域內(nèi)的折射率，從而優(yōu)化系統(tǒng)性能。

通過(guò)現(xiàn)場(chǎng)梯度下降方法，光學(xué)系統(tǒng)可以逐步調(diào)整其參數(shù)，直至達(dá)到最優(yōu)狀態(tài)。

原文將上述全前向模式的梯度下降方法（替代反向傳播）用方程最終表示為：

一種光學(xué)神經(jīng)網(wǎng)絡(luò)訓(xùn)練方法

作為一種光學(xué)神經(jīng)網(wǎng)絡(luò)訓(xùn)練的方法，F(xiàn)FM 有以下優(yōu)勢(shì)：

與理想模型相當(dāng)?shù)臏?zhǔn)確率

使用 FFM 可以在自由空間光學(xué)神經(jīng)網(wǎng)絡(luò)（Optical Neural Network，ONN）上實(shí)現(xiàn)有效的自訓(xùn)練過(guò)程。

要說(shuō)明這個(gè)結(jié)論，研究人員首先用一個(gè)單層的ONN 在基準(zhǔn)數(shù)據(jù)集上進(jìn)行了對(duì)象分類(lèi)訓(xùn)練（a）。

具體來(lái)說(shuō)，他們用了一些手寫(xiě)數(shù)字的圖片（MNIST 數(shù)據(jù)集）來(lái)訓(xùn)練這個(gè)系統(tǒng)，然后將結(jié)果進(jìn)行了可視化（b）。

結(jié)果顯示，通過(guò) FFM 學(xué)習(xí)訓(xùn)練的 ONN 在實(shí)驗(yàn)光場(chǎng)與理論光場(chǎng)之間相似性極高（SSIM 超過(guò) 0.97）。

換句話(huà)說(shuō)，它學(xué)習(xí)得非常好，幾乎能夠完美復(fù)制給它的示例。

However, researchers also remind:

Due to imperfections in the system, the theoretically calculated light fields and gradients cannot fully accurately reflect actual physical phenomena.

Next, the researchers used more complex images (Fashion-MNIST dataset) to train the system to recognize different fashion items.

In the beginning, when the number of layers increased from 2 to 8, the average accuracy of the computer-trained network was almost half of the theoretical accuracy.

With the FFM learning method, the network accuracy of the system has been increased to 92.5%, which is close to the theoretical value.

This shows that as the number of network layers increases, the performance of the network trained by traditional methods decreases, while FFM learning can maintain high accuracy.

At the same time, the performance of ONN can be further improved by incorporating nonlinear activation into FFM learning. In experiments, nonlinear FFM learning was able to improve classification accuracy from 90.4% to 93.0%.

Research further proves that by batch training non-linear ONN, the error propagation process can be simplified and the training time only increases by 1 to 1.7 times.

High-resolution focusing capability

FFM can also achieve high-quality imaging in practical applications, achieving resolution close to the physical limit even in complex scattering environments.

First of all, when light waves enter a scattering medium (such as fog, smoke or biological tissue, etc.), focusing will become complicated, but the propagation of light waves in the medium often maintains a certain symmetry.

FFM takes advantage of this symmetry by optimizing the propagation path and phase of light waves to reduce the negative impact of scattering effects on focusing.

The effect is also very significant. Figure b shows the comparison of the two optimization methods, FFM and PSO (Particle Swarm Optimization).

Specifically, the experiment used two scattering media, one is a random phase plate (Scatterer-I) and the other is transparent tape (Scatterer-II).

In both media, FFM achieved convergence (finding the optimal solution faster) after only 25 design iterations, with convergence loss values ??of 1.84 and 2.07 respectively (lower is better performance).

The PSO method requires at least 400 design iterations to reach convergence, and the loss values ??at final convergence are 2.01 and 2.15.

At the same time, Figure c shows that FFM is able to continuously optimize itself, and the focus it is designed to gradually evolve and converge from an initial random distribution to a tight focus.

Within a design area of ??3.2 mm × 3.2 mm, the researchers further uniformly sampled the FFM and PSO optimized foci and compared their FWHM (full width at half maximum) and PSNR (peak signal to noise ratio).

The results show that FFM has higher focusing accuracy and better imaging quality.

Figure e further evaluates the performance of the designed focus array when scanning a resolution map located behind a scattering medium.

The results are surprising. The focus size of the FFM design is close to the diffraction limit of 64.5 m, which is the theoretical highest resolution standard for optical imaging.

Able to parallelly image objects outside the line of sight

Since it is so powerful in scattering media, the researchers also tried non-line-of-sight (NLOS) scenarios, where objects are hidden from sight.

FFM exploits the spatial symmetry of the light path from the hidden object to the observer, which allows the system to reconstruct and analyze dynamic hidden objects in the field in an all-optical manner.

By designing the input wavefront, FFM is able to simultaneously project all meshes in the object to their target positions, achieving parallel recovery of hidden objects.

The letter-shaped hidden chromium targets "T", "H" and "U" were used in the experiment, and the exposure time (1 millisecond) and optical power (0.20 mW) were set to achieve rapid imaging of these dynamic targets.

The results show that without the FFM designed wavefront, the image will be severely distorted. While the FFM-designed wavefront was able to recover the shapes of all three letters, the SSIM (structural similarity index) reached 1.0, indicating a high degree of similarity to the original image.

Further, compared with artificial neural network (ANN) in terms of photon efficiency and classification performance, FFM significantly outperforms ANN, especially under low-photon conditions.

Specifically, in situations where the number of photons is limited (such as many reflective or highly diffuse surfaces), FFM is able to adaptively correct wavefront distortion and require fewer photons for accurate classification.

Automatic search for outliers in non-Hermitian systems

FFM methods are not only applicable to free-space optical systems, but can also be extended to the self-design of integrated photonic systems.

The researchers constructed an integrated neural network (a) using symmetric photonic cores configured in series and parallel.

In the experiment, the symmetric core was configured with a variable optical attenuator (VOA) through different levels of injection current to achieve different attenuation coefficients to simulate different weights.

In Figure c, the fidelity of the programmed matrix values ??in the symmetric core is very high, with standard deviations of time drift of 0.012%, 0.012% and 0.010% respectively, indicating that the matrix values ??are very stable.

And, the researchers visualized the error for each layer. Comparing the experimental gradient with the theoretical simulation value, the average deviation is 3.5%.

After approximately 100 iterations (epochs), the network reaches convergence.

Experimental results show that under three different symmetry ratio configurations (1.0, 0.75 or 0.5), the classification accuracy of the network is 94.7%, 89.2% and 89.0% respectively.

The classification accuracy obtained by using the neural network using the FFM method is 94.2%, 89.2% and 88.7%.

In contrast, if traditional computer simulation methods are used to design the network, the classification accuracy of the experiment will be lower, respectively 71.7%, 65.8% and 55.0%.

Finally, the researchers also demonstrated that FFM can self-design non-Hermitian systems and achieve traversal of singular points without the need for physical models through numerical simulation.

Non-Hermitian system is a concept in physics, which involves systems in fields such as quantum mechanics and optics, which do not satisfy Hermitian conditions.

Hermitian properties are related to the symmetry of the system and the real number of energy. Non-Hermitian systems do not meet these conditions. They may have some special physical phenomena, such as exceptional points (Exceptional Points), which are the dynamics of the system. Where learning behavior undergoes strange changes at certain points.

To summarize the full article, FFM is a method to implement computationally intensive training processes on physical systems, capable of efficiently executing most machine learning operations in parallel.

For more detailed experimental settings and data set preparation process, please refer to the original article.

Code:

https://zenodo.org/records/10820584

Original text of "Nature":

https://www.nature.com/articles/s41586-024-07687-4

The above is the detailed content of Tsinghua Optics AI appears in Nature! Physical neural network, backpropagation is no longer needed. For more information, please follow other related articles on the PHP Chinese website!