GoKawiilNext, we demonstrate that RL is not only able to optimize the system performance, but also steer the control policy in the presence of drift. This is achieved by the entropy regularization technique45, which ensures that the policy distribution never becomes deterministic, enabling the agent to continuously explore and adapt to changes in a non-stationary environment. To systematically study this effect, we inject artificial drift on several control parameters simultaneously, as shown in Fig. 4a (red symbols). We choose different temporal drift profiles (step-like, sinusoidal and stroboscopic), different control parameters (CZ coupling strength, XY pulse amplitude and frequency) and different locations of drifting gates on the qubit grid.
Fig. 4: Demonstration of RL steering. Full size image a, The data qubits (yellow diamonds) and measure qubits (panels with data) are arranged in the layout of a distance-5 surface code. We inject artificial drift with various temporal profiles on the gates indicated with red shapes (circles, diamonds and bars) and observe elevated error-detection signals where expected (coloured background). The detection rate associated with each measure qubit is normalized for visualization to remove the effect of the natural system drift. Performance of the fixed control policy (maroon) degrades over time because of the injected drift, whereas RL steering (blue) stabilizes and maintains the error-detection rate (EDR) below its initial level (white lines). b, Time dependence of the injected drift in system control parameters (dashed) and RL steering (solid). An exponential fit of RL response to step-like drift in XY pulse amplitude yields the characteristic learning time of 130 epochs. c, Periodic evaluation of the logical performance indicates that RL steering of the system significantly suppresses and stabilizes the LER, see main text. Incorporating the decoder steering (black) further improves these results.
Following the already-described methodology, we start by calibrating the control policy at t = 0. The performance of this policy (maroon) degrades over time because of the injected drift, as the fixed values of the control parameters become ‘outdated’, leading to additional errors. These control errors lead to an elevated detection event signal, which appears exactly in those detectors in which it is expected based on the constructed factor graph, see tiles highlighted with coloured background in Fig. 4a. In contrast to a fixed control policy, RL steering (blue) maintains a significantly suppressed rate of error-detection events that consistently remains below the initial level (white line), except for brief moments when the drift is too fast. In Fig. 4b, we show the evolution of the learned control parameters. The recovery from a step-like drift in the XY pulse amplitude (red circle) allows us to estimate the response time of the steering process of about 130 epochs. This also sets the time scale for the policy lag in the case of slow continuous drift, as in XY pulse frequency (red diamond).
To confirm that suppression of detection events is not due to hindered detection capability but is due to suppressed errors, we evaluate the logical performance in Fig. 4c. Compared with the fixed policy, we find on average a 24% reduction of LER and a 2.4× improvement of its stability (quantified here by the standard deviation of the LER distribution). These figures of merit further improve to 31% and 3.5×, respectively, by additionally steering the decoder parameters. Decoder steering is achieved within the same RL framework by reweighting the matching graph as described in ref. 46. Although the steering of the classical controller is done via the surrogate objective C that relies exclusively on the error-detection probabilities, our decoder steering process relies on LER estimation, which is not straightforwardly scalable to the real-time setting. However, alternative approaches to decoder steering have been proposed47,48,49 that, in principle, do not suffer from this limitation.
We also analysed the RL performance under natural system drift, which arises from sources ranging from material defects near the quantum system50 to temperature fluctuations in the classical control instruments and, unlike our previous demonstration, rarely has a simple time dependence. Fourier analysis of multiple experimental RL runs from Fig. 3a shows that the effect of steering can be understood as a filter that provides about 4 dB of suppression of low-frequency LER fluctuations originating from these natural sources (Supplementary Information section III).
We have thus far demonstrated that the mean μ(t) of the Gaussian policy distribution learns to track the optimal policy over time in the presence of drift. As a result, the learned policy μ(t) outperforms the fixed policy, with lower EDR and LER. However, as the learning process requires exploration of the parameter space, the algorithm inevitably samples policy candidates whose performance is worse than that of the μ(t) policy. This ‘exploration noise’ is irrelevant in our experimental setting in which RL steering relies on repeated executions of a short logical algorithm and the quantum state is independently re-prepared in every shot. However, in the future, this steering must be done in real time during the single-shot execution of a long logical algorithm. In that case, the exploration noise, although necessary for learning, will be detrimental to the performance of the logical algorithm. Generally, balancing the exploration of parameter space and the exploitation of the learned policy μ(t) is a central challenge in many applications of RL51. In our case, the favourable resolution of the exploration–exploitation trade-off would mean that the aggregate performance of all sampled policy candidates, most of which are worse than μ(t), is still better than the performance without RL steering.
To study this trade-off, we conducted numerical simulations of real-time steering of the distance-3 surface code subject to sinusoidal parameter drift at different frequencies (Fig. 5a and Supplementary Information section VIA). We count the total number of error-detection events generated in a 1.8 × 109-cycle window of QEC, and normalize it so that level 1 corresponds to the performance of the optimal policy (known in the simulation), and level 0 corresponds to the performance of a fixed policy (optimal at t = 0). We control the exploration–exploitation trade-off by changing the amount of entropy regularization45 in our RL algorithm. Our findings indicate that there is a critical drift frequency, about 1/150 epochs, below which the system becomes real-time steerable: the performance with exploration noise is better than the performance of a fixed policy. This critical frequency is consistent with the response time of the learning algorithm observed in the experiment (Fig. 4b). When drift is too fast, the real-time steering is not able to keep up—this drift must be mitigated at the hardware level. In particular, correlated drift caused by rare high-energy particle impacts in superconducting devices8,41 typically equilibrates on a much shorter time scale and is not real-time steerable in our current implementation. By contrast, at low drift frequencies, the exploration and exploitation can be successfully balanced to closely approach the performance of the optimal policy at all times.
Fig. 5: Real-time steering and scaling simulations. Full size image a, Normalized improvement (colour) of error-detection rate in the real-time steering simulation of the distance-3 surface code subject to sinusoidal drift at different frequencies. Level 1 indicates the performance of the optimal policy. Isoline at level 0 demarcates the boundary beyond which real-time steering results in better performance than a fixed policy, approaching the performance of an optimal policy in the regime of slow drift. b, Simulation of scalability of RL control of large surface codes. The algorithm learns the parameters of single-qubit and CZ gates, with 30 control parameters per gate, amounting to almost 40,000 control parameters in total for the distance-15 code. During the learning process, the LER reduces over time (colour) until it reaches the floor (red bars) set by the irreducible physical error rates and characterized by the optimal error suppression factor Λ*. c, Point estimates of Λ at every code distance and learning epoch from b confirm that the speed, ∂ t Λ/Λ* × 102, at which the error suppression factor approaches the local optimum, is proportional to the distance from the optimum, 1 − Λ/Λ*. The convergence rate γ (see main text) is independent of the system size but depends on the number of control parameters per gate, with three beams corresponding to 1, 10 and 30 parameters, and the linear fits (red) indicating the convergence rates.
Thus, our simulations establish that RL is able to effectively use the information concealed in the error-detection events to calibrate the system while continuing the logical computation. This ability offers a substantial advantage over approaches based on the synthesis of traditional calibration and code deformation21, because RL steering does not introduce any resource overhead.