DNS Zone Propagation Time
Please note a slowdown on the DNS Propagation Time on all DNS Servers
Following all the necessary information.
* Start time: Thu Jun 10 2021 17:30:00 Europe/Paris
* Severity level : Medium
* User impact : Slowdowns on DNS zone updates
* Technical impact (DC/geographical area) : All DNS Servers (except CA)
* Root cause : Storage
* Scope of impact : All DNS Servers
* Time resolution : Unknown ETA
* Current status : 90 %
* On-going actions : Storage
* Workaround : NA
Date: 2021-06-12 21:03:29 UTC
DNSSEC operations are now all done
Date: 2021-06-12 08:42:56 UTC
DNS Propagation Time in now back to normal.
We still have some DNSSEC operations in error due to this task.
All have been relaunched. It might be finish by the end of the day.
Date: 2021-06-11 22:58:41 UTC
Zone updates are still on going, > 80% has been refreshed.
Date: 2021-06-11 20:20:26 UTC
Zone updates are still on going, 1/3 has been refreshed
Date: 2021-06-11 18:15:37 UTC
The new storage engine has been installed
Robots are now running and zone updates are being deploy
Everything will be back to normal within the next few hours
Date: 2021-06-11 16:53:53 UTC
Robots are now stopped and a new storage is being deployed
Once this is done we will be able to restart the zone update
Date: 2021-06-11 15:15:07 UTC
We found an issue on exclusive locks on the storage part. We try to move to a new type of storage to recover full speed.
In the meantime, we are running without lock to speed up the dns update zone.
Date: 2021-06-11 08:24:25 UTC
We replaced the zone file storage last night, but we have still some issue with I/O, so dns propagation is very slow for now.
We are still troubleshooting the issue with the storage team.
This incident affected: DNS || DNS Hosting (dns, dns10, dns11, dns12, dns13, dns14, dns15, dns16, dns17, dns18, dns19, dns20, dns100, dns101, dns102, dns103, dns104, dns105, dns106, dns107, dns108, dns109, dns110, dns111, dns112, dns113, dns-tn, dns10-ca).