Time-synchronization errors

Incident Report for INFO

Postmortem

Incident:
A central time (NTP) server within the INFO network lost time-synchronization.

Consequence:
Servers that were synchronizing time with the time announced by the failing central time server started to drift too far away from the correct time, applications dependent on the correct time on the server and/or client (typically 2-factor authentication systems) started to fail.

Environment:
All customer servers hosted at our data center facility and managed within the INFO network.

Cause:
The (virtual) central time server was moved to another node within our virtualization cluster, after this migration the NTP (network time protocol) process became unresponsive.

Remedy:
The central time server was restarted manually. This brought the NTP process back again.

Solution:
The automatic optimization process within our virtualization cluster that was responsible for moving the central time server within the cluster to another node has been configured to no longer move the central time server.

Remarks:
Right after the failure on the central time server, the system time on connected (virtual) servers started to drift away from the correct time by about 5 minutes per hour. The central time server started announcing the proper time again right after the restart, but for all connected servers the process of drifting back to the proper system time is the same gradual process and complete recovery might take a few hours for some servers. Therefore some critical servers that are used for 2-factor authentication have been synchronized manually where possible.

Follow-up:
The current time server setup will be reviewed by our engineers and they will investigate the possibilities of making the current NTP configuration within our network more resistant to failures.

Posted Nov 04, 2020 - 12:00 CET

Resolved

Some servers are still on their way to getting the appropriate system time again, and issues may still be happening as a consequence of that. However, the servers are slowly recovering surely. We will continue to monitor this behavior but we will now close this incident as the cause has been rectified. We will provide a "post mortem" report on this incident with some more details on the follow-up and steps we are taking to make sure this incident does not repeat itself in the future.

Posted Nov 04, 2020 - 11:35 CET

Monitoring

We have put the NTP server that was announcing a faulty system time back into sync. The connected servers however, are now "drifting" back to the accurate system time because of this. This process cannot be rushed and moves at a slow pace in order to prevent even bigger errors.

We suspect at this point all system times will be correct again within 2-3 hours. We will continue to monitor this process and where possible, accelerate this as much as we can.

This will primarily affect things like 2-factor authentication on a platform, since those processes depend heavily on time-stamps. We will continue to provide updates when needed.

Posted Nov 04, 2020 - 10:12 CET

Update

We are continuing to work on a fix for this issue.

Posted Nov 04, 2020 - 09:15 CET

Identified

Our engineers have identified an issue with system time synchronization on almost all servers at INFO causing - among other things - login-failures on multiple systems and applications.

We are working hard to bring all system times on our servers back into sync as fast as possible.

Posted Nov 04, 2020 - 09:08 CET

This incident affected: Infrastructure as a Service (IaaS) (VMware).