zkSync Developers Addresses Technical ‘Subtle Bug’ Promptly; Releases Updated Software

• On December 25, zkSync Era faced a hiccup triggered by an automated safety procedure.
• The root cause of the problem lay in a subtle bug related to state updates—a critical element within the protocol. 
• To ensure network-wide security, an updated version of the node software was promptly released for deployment by other node operators.

On December 25, zkSync Era faced a hiccup triggered by an automated safety procedure. While the issue has been successfully addressed, the zkSync developer team is keen on sharing the intricate details of the incident with the wider community.

The root cause of the problem lay in a subtle bug related to state updates—a critical element within the protocol. The bug, tied to how the operator computed and transmitted updates to Layer 1 (L1), plunged the network into a defensive mode. Notably, the protocol’s computations were accurate, but a flaw in the operator’s process led to this unexpected divergence.

Once pinpointed, the development team acted swiftly to resolve the bug, allowing the sequencer to resume its transaction processing duties. To ensure network-wide security, an updated version of the node software was promptly released for deployment by other node operators.

The defensive posture, initially implemented over a year ago in anticipation of Era’s mainnet launch, proved to be surplus to requirements. The system could have continued to operate during the issue resolution period, thanks to a built-in 24-hour execution delay designed for such contingencies.

However, the resolution process encountered delays due to the holiday season, with only on-call engineers available. This incident has sparked discussions within the team about sunsetting this safety procedure and refining the network’s holiday escalation plan, topics that will be prioritized in the upcoming year.

The zkSync team extends sincere apologies to affected developers and users for any inconvenience caused. The community’s patience and offers of assistance during this incident are deeply appreciated.