OVHcloud Web Hosting Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
max_connections on SharedSQL
Incident
Report for Web Cloud
Resolved
We encounter errors about \"max_user_connections\" that look illegitimate on P19 SharedSQL. Investigation in progress.
---
Nous rencontrons des erreurs de \"max_user_connections\" qui semblent illégitimes sur les SharedSQL de P19. Nous investigons.
Update(s):
Date: 2018-12-18 09:12:18 UTC
The patch is deployed on all SQL servers.
We keep an eye on SQL latency to ensure the patch resolve all isssues.
-----------
Le patch est déployé sur l'ensemble des serveurs SQL.
Nous continuons de surveiller les latences SQL pour vérifier le bon fonctionnement de ce patch
Date: 2018-12-17 10:33:37 UTC
mysql55-1.abuse.ha.ovh.net
mysql55-125.bdb.ha.ovh.net
mysql55-137.bdb.ha.ovh.net
mysql55-147.bdb.ha.ovh.net
mysql55-14.bdb.ha.ovh.net
mysql55-162.bdb.ha.ovh.net
mysql55-18.bdb.ha.ovh.net
mysql55-208.bdb.ha.ovh.net
mysql55-216.bdb.ha.ovh.net
mysql55-223.bdb.ha.ovh.net
mysql55-229.bdb.ha.ovh.net
mysql55-235.bdb.ha.ovh.net
mysql55-245.bdb.ha.ovh.net
mysql55-247.bdb.ha.ovh.net
mysql55-261.bdb.ha.ovh.net
mysql55-35.bdb.ha.ovh.net
mysql55-43.bdb.ha.ovh.net
mysql55-46.bdb.ha.ovh.net
mysql55-48.bdb.ha.ovh.net
mysql55-72.bdb.ha.ovh.net
mysql55-73.bdb.ha.ovh.net
mysql55-74.bdb.ha.ovh.net
mysql55-120.business.ha.ovh.net
mysql55-153.business.ha.ovh.net
mysql55-155.business.ha.ovh.net
mysql55-158.business.ha.ovh.net
mysql55-170.business.ha.ovh.net
mysql55-114.perso.ha.ovh.net
mysql55-118.perso.ha.ovh.net
mysql55-138.perso.ha.ovh.net
mysql55-146.perso.ha.ovh.net
mysql55-156.perso.ha.ovh.net
mysql55-166.perso.ha.ovh.net
mysql55-173.perso.ha.ovh.net
mysql55-177.perso.ha.ovh.net
mysql55-178.perso.ha.ovh.net
mysql55-17.perso.ha.ovh.net
mysql55-18.perso.ha.ovh.net
mysql55-190.perso.ha.ovh.net
mysql55-194.perso.ha.ovh.net
mysql55-197.perso.ha.ovh.net
mysql55-19.perso.ha.ovh.net
mysql55-218.perso.ha.ovh.net
mysql55-230.perso.ha.ovh.net
mysql55-239.perso.ha.ovh.net
mysql55-23.perso.ha.ovh.net
mysql55-242.perso.ha.ovh.net
mysql55-251.perso.ha.ovh.net
mysql55-252.perso.ha.ovh.net
mysql55-253.perso.ha.ovh.net
mysql55-259.perso.ha.ovh.net
mysql55-26.perso.ha.ovh.net
mysql55-28.perso.ha.ovh.net
mysql55-34.perso.ha.ovh.net
mysql55-38.perso.ha.ovh.net
mysql55-41.perso.ha.ovh.net
mysql55-43.perso.ha.ovh.net
mysql55-46.perso.ha.ovh.net
mysql55-52.perso.ha.ovh.net
mysql55-53.perso.ha.ovh.net
mysql55-57.perso.ha.ovh.net
mysql55-65.perso.ha.ovh.net
mysql55-7.perso.ha.ovh.net
mysql55-9.perso.ha.ovh.net
mysql55-136.premium.ha.ovh.net
mysql55-102.pro.ha.ovh.net
mysql55-16.bdb.ha.ovh.net
mysql55-190.bdb.ha.ovh.net
mysql55-225.bdb.ha.ovh.net
mysql55-226.bdb.ha.ovh.net
mysql55-106.bdb.ha.ovh.net
mysql55-62.bdb.ha.ovh.net
mysql55-68.bdb.ha.ovh.net
mysql55-116.business.ha.ovh.net
mysql55-103.perso.ha.ovh.net
mysql55-109.perso.ha.ovh.net
mysql55-110.perso.ha.ovh.net
mysql55-122.perso.ha.ovh.net
mysql55-127.perso.ha.ovh.net
mysql55-157.perso.ha.ovh.net
mysql55-159.perso.ha.ovh.net
mysql55-167.perso.ha.ovh.net
mysql55-171.perso.ha.ovh.net
mysql55-179.perso.ha.ovh.net
mysql55-181.perso.ha.ovh.net
mysql55-211.perso.ha.ovh.net
mysql55-223.perso.ha.ovh.net
mysql55-237.perso.ha.ovh.net
mysql55-40.perso.ha.ovh.net
mysql55-10.pro.ha.ovh.net
mysql55-115.pro.ha.ovh.net
mysql55-137.pro.ha.ovh.net
mysql55-156.pro.ha.ovh.net
mysql55-182.pro.ha.ovh.net
mysql55-23.pro.ha.ovh.net
mysql55-111.pro.ha.ovh.net
mysql55-135.pro.ha.ovh.net
mysql55-149.pro.ha.ovh.net
mysql55-175.pro.ha.ovh.net
mysql55-181.pro.ha.ovh.net
mysql55-109.bdb
mysql55-111.bdb
mysql55-121.bdb
mysql55-122.bdb
mysql55-132.bdb
mysql55-150.bdb
mysql55-157.bdb
mysql55-160.bdb
mysql55-17.bdb
mysql55-214.bdb
mysql55-227.bdb
mysql55-233.bdb
mysql55-234.bdb
mysql55-238.bdb
mysql55-134.business
mysql55-135.business
mysql55-106.perso
mysql55-140.perso
mysql55-168.perso
mysql55-217.perso
mysql55-224.perso
mysql55-226.perso
mysql55-227.perso
mysql55-22.perso
mysql55-250.perso
mysql55-48.perso
mysql55-4.perso
mysql55-55.perso
mysql55-105.pro
mysql55-110.pro
mysql55-120.pro
mysql55-131.pro
mysql55-133.pro
mysql55-136.pro
mysql55-146.pro
mysql55-148.pro
mysql55-153.pro
mysql55-154.pro
mysql55-15.pro
mysql55-164.pro
mysql55-16.pro
mysql55-183.pro
mysql55-20.pro
mysql55-21.pro
mysql55-27.pro
mysql55-33.pro
mysql55-34.pro
mysql55-3.pro
mysql55-4.pro
mysql55-7.pro
Date: 2018-12-17 10:33:16 UTC
Le second correctif a été déployé sur les serveurs les plus impactés.
--
The second patch has been deployed on the most impacted servers.
Date: 2018-12-14 17:00:34 UTC
Un deuxième correctif est en cours de déploiement
---
A second patch is being deployed
Date: 2018-12-11 08:47:01 UTC
All had been deployed.
---
Tout a été déployé.
Date: 2018-12-10 23:17:42 UTC
Second part (50%) had been deployed.
---
La seconde partie (50%) a été déployée.
Date: 2018-12-10 17:32:54 UTC
First part had been deployed.
---
La première partie a été déployée.
Date: 2018-12-10 16:43:55 UTC
Patch will be deployed in a couple of minutes.
---
Le patch va être déployé dans quelques instants.
Date: 2018-12-10 15:24:29 UTC
Patch had been deployed everywhere. Another one is being tested to ensure sustainability of the changes.
---
Le patch a été déployé partout. Un autre est en cours de tests afin d'assurer la pérénité de ces changements.
Date: 2018-12-10 14:34:36 UTC
ETA: 16:30 french time
Date: 2018-12-10 13:50:22 UTC
About half of the servers are now patched.
---
Le patch est maintenant déployé sur la moitié des serveurs.
Date: 2018-12-10 08:31:58 UTC
All the servers will be patched in a couple of minutes.
---
L'ensemble des serveurs va être patché dans quelques minutes.
Date: 2018-12-08 17:41:45 UTC
We have just patched additional servers to fix some residual max_user_connections reached:
mysql55-224.perso.ha.ovh.net
mysql55-142.perso.ha.ovh.net
mysql55-41.perso.ha.ovh.net
mysql55-205.perso.ha.ovh.net
mysql55-75.bdb.ha.ovh.net
mysql55-108.perso.ha.ovh.net
Task is closed.
Date: 2018-12-08 15:01:22 UTC
Saturday 10:10
Below the full list of servers with users reaching max_user_connections:
mysql55-110.perso.ha.ovh.net: 50
mysql55-34.perso.ha.ovh.net: 56
mysql55-196.perso.ha.ovh.net: 3
mysql55-40.perso.ha.ovh.net: 5
mysql55-211.perso.ha.ovh.net: 4
mysql55-58.perso.ha.ovh.net: 82
mysql55-239.bdb.ha.ovh.net: 21
mysql55-237.perso.ha.ovh.net: 7
mysql55-157.perso.ha.ovh.net: 41
mysql55-116.business.ha.ovh.net: 23
mysql55-165.perso.ha.ovh.net: 37
mysql55-167.perso.ha.ovh.net: 216
mysql55-253.perso.ha.ovh.net: 61
mysql55-171.perso.ha.ovh.net: 39
mysql55-127.perso.ha.ovh.net: 61
mysql55-177.perso.ha.ovh.net: 25
mysql55-14.bdb.ha.ovh.net: 4
mysql55-55.perso.ha.ovh.net: 4
mysql55-15.perso.ha.ovh.net: 174
mysql55-106.perso.ha.ovh.net: 117
mysql55-162.pro.ha.ovh.net: 17
mysql55-255.perso.ha.ovh.net: 127
mysql55-121.perso.ha.ovh.net: 126
mysql55-112.perso.ha.ovh.net: 19
mysql55-128.perso: ?
The appropriate patch has been sucessfully deployed on these servers.
Saturday 12:05:
No more users reaching max_user_connections on theirs DB.
To limit unwanted side effects, the patch will be deployed on all servers from Monday.
Date: 2018-12-07 15:03:24 UTC
Those databases errors widely result in 500 errors on your website.
The deployment of the ProxySQL (see http://travaux.ovh.net/?do=details&id=35499) and a tiny error on few customers resulted in a snow-ball effect. 2 causes resulting in a problem, each of them, independently, are not a problem at all. That's why the diagnose was hard, and that's why the change passed our tests.
---
Ces erreurs de bases de données aboutissent généralement à une erreur 500 sur votre site web.
Le déploiement des ProxySQL (cf. http://travaux.ovh.net/?do=details&id=35499) ainsi qu'une petite erreur sur quelques clients ont entraîné un effet boule de neige. 2 causes qui aboutissent à un probleme, mais chacune d'elles, prises indépendamment, n'en provoquent pas. C'est pourquoi le problème était dur à diagnostiquer, c'est également pourquoi le changement a passé tous nos tests.
---
Nous rencontrons des erreurs de \"max_user_connections\" qui semblent illégitimes sur les SharedSQL de P19. Nous investigons.
Update(s):
Date: 2018-12-18 09:12:18 UTC
The patch is deployed on all SQL servers.
We keep an eye on SQL latency to ensure the patch resolve all isssues.
-----------
Le patch est déployé sur l'ensemble des serveurs SQL.
Nous continuons de surveiller les latences SQL pour vérifier le bon fonctionnement de ce patch
Date: 2018-12-17 10:33:37 UTC
mysql55-1.abuse.ha.ovh.net
mysql55-125.bdb.ha.ovh.net
mysql55-137.bdb.ha.ovh.net
mysql55-147.bdb.ha.ovh.net
mysql55-14.bdb.ha.ovh.net
mysql55-162.bdb.ha.ovh.net
mysql55-18.bdb.ha.ovh.net
mysql55-208.bdb.ha.ovh.net
mysql55-216.bdb.ha.ovh.net
mysql55-223.bdb.ha.ovh.net
mysql55-229.bdb.ha.ovh.net
mysql55-235.bdb.ha.ovh.net
mysql55-245.bdb.ha.ovh.net
mysql55-247.bdb.ha.ovh.net
mysql55-261.bdb.ha.ovh.net
mysql55-35.bdb.ha.ovh.net
mysql55-43.bdb.ha.ovh.net
mysql55-46.bdb.ha.ovh.net
mysql55-48.bdb.ha.ovh.net
mysql55-72.bdb.ha.ovh.net
mysql55-73.bdb.ha.ovh.net
mysql55-74.bdb.ha.ovh.net
mysql55-120.business.ha.ovh.net
mysql55-153.business.ha.ovh.net
mysql55-155.business.ha.ovh.net
mysql55-158.business.ha.ovh.net
mysql55-170.business.ha.ovh.net
mysql55-114.perso.ha.ovh.net
mysql55-118.perso.ha.ovh.net
mysql55-138.perso.ha.ovh.net
mysql55-146.perso.ha.ovh.net
mysql55-156.perso.ha.ovh.net
mysql55-166.perso.ha.ovh.net
mysql55-173.perso.ha.ovh.net
mysql55-177.perso.ha.ovh.net
mysql55-178.perso.ha.ovh.net
mysql55-17.perso.ha.ovh.net
mysql55-18.perso.ha.ovh.net
mysql55-190.perso.ha.ovh.net
mysql55-194.perso.ha.ovh.net
mysql55-197.perso.ha.ovh.net
mysql55-19.perso.ha.ovh.net
mysql55-218.perso.ha.ovh.net
mysql55-230.perso.ha.ovh.net
mysql55-239.perso.ha.ovh.net
mysql55-23.perso.ha.ovh.net
mysql55-242.perso.ha.ovh.net
mysql55-251.perso.ha.ovh.net
mysql55-252.perso.ha.ovh.net
mysql55-253.perso.ha.ovh.net
mysql55-259.perso.ha.ovh.net
mysql55-26.perso.ha.ovh.net
mysql55-28.perso.ha.ovh.net
mysql55-34.perso.ha.ovh.net
mysql55-38.perso.ha.ovh.net
mysql55-41.perso.ha.ovh.net
mysql55-43.perso.ha.ovh.net
mysql55-46.perso.ha.ovh.net
mysql55-52.perso.ha.ovh.net
mysql55-53.perso.ha.ovh.net
mysql55-57.perso.ha.ovh.net
mysql55-65.perso.ha.ovh.net
mysql55-7.perso.ha.ovh.net
mysql55-9.perso.ha.ovh.net
mysql55-136.premium.ha.ovh.net
mysql55-102.pro.ha.ovh.net
mysql55-16.bdb.ha.ovh.net
mysql55-190.bdb.ha.ovh.net
mysql55-225.bdb.ha.ovh.net
mysql55-226.bdb.ha.ovh.net
mysql55-106.bdb.ha.ovh.net
mysql55-62.bdb.ha.ovh.net
mysql55-68.bdb.ha.ovh.net
mysql55-116.business.ha.ovh.net
mysql55-103.perso.ha.ovh.net
mysql55-109.perso.ha.ovh.net
mysql55-110.perso.ha.ovh.net
mysql55-122.perso.ha.ovh.net
mysql55-127.perso.ha.ovh.net
mysql55-157.perso.ha.ovh.net
mysql55-159.perso.ha.ovh.net
mysql55-167.perso.ha.ovh.net
mysql55-171.perso.ha.ovh.net
mysql55-179.perso.ha.ovh.net
mysql55-181.perso.ha.ovh.net
mysql55-211.perso.ha.ovh.net
mysql55-223.perso.ha.ovh.net
mysql55-237.perso.ha.ovh.net
mysql55-40.perso.ha.ovh.net
mysql55-10.pro.ha.ovh.net
mysql55-115.pro.ha.ovh.net
mysql55-137.pro.ha.ovh.net
mysql55-156.pro.ha.ovh.net
mysql55-182.pro.ha.ovh.net
mysql55-23.pro.ha.ovh.net
mysql55-111.pro.ha.ovh.net
mysql55-135.pro.ha.ovh.net
mysql55-149.pro.ha.ovh.net
mysql55-175.pro.ha.ovh.net
mysql55-181.pro.ha.ovh.net
mysql55-109.bdb
mysql55-111.bdb
mysql55-121.bdb
mysql55-122.bdb
mysql55-132.bdb
mysql55-150.bdb
mysql55-157.bdb
mysql55-160.bdb
mysql55-17.bdb
mysql55-214.bdb
mysql55-227.bdb
mysql55-233.bdb
mysql55-234.bdb
mysql55-238.bdb
mysql55-134.business
mysql55-135.business
mysql55-106.perso
mysql55-140.perso
mysql55-168.perso
mysql55-217.perso
mysql55-224.perso
mysql55-226.perso
mysql55-227.perso
mysql55-22.perso
mysql55-250.perso
mysql55-48.perso
mysql55-4.perso
mysql55-55.perso
mysql55-105.pro
mysql55-110.pro
mysql55-120.pro
mysql55-131.pro
mysql55-133.pro
mysql55-136.pro
mysql55-146.pro
mysql55-148.pro
mysql55-153.pro
mysql55-154.pro
mysql55-15.pro
mysql55-164.pro
mysql55-16.pro
mysql55-183.pro
mysql55-20.pro
mysql55-21.pro
mysql55-27.pro
mysql55-33.pro
mysql55-34.pro
mysql55-3.pro
mysql55-4.pro
mysql55-7.pro
Date: 2018-12-17 10:33:16 UTC
Le second correctif a été déployé sur les serveurs les plus impactés.
--
The second patch has been deployed on the most impacted servers.
Date: 2018-12-14 17:00:34 UTC
Un deuxième correctif est en cours de déploiement
---
A second patch is being deployed
Date: 2018-12-11 08:47:01 UTC
All had been deployed.
---
Tout a été déployé.
Date: 2018-12-10 23:17:42 UTC
Second part (50%) had been deployed.
---
La seconde partie (50%) a été déployée.
Date: 2018-12-10 17:32:54 UTC
First part had been deployed.
---
La première partie a été déployée.
Date: 2018-12-10 16:43:55 UTC
Patch will be deployed in a couple of minutes.
---
Le patch va être déployé dans quelques instants.
Date: 2018-12-10 15:24:29 UTC
Patch had been deployed everywhere. Another one is being tested to ensure sustainability of the changes.
---
Le patch a été déployé partout. Un autre est en cours de tests afin d'assurer la pérénité de ces changements.
Date: 2018-12-10 14:34:36 UTC
ETA: 16:30 french time
Date: 2018-12-10 13:50:22 UTC
About half of the servers are now patched.
---
Le patch est maintenant déployé sur la moitié des serveurs.
Date: 2018-12-10 08:31:58 UTC
All the servers will be patched in a couple of minutes.
---
L'ensemble des serveurs va être patché dans quelques minutes.
Date: 2018-12-08 17:41:45 UTC
We have just patched additional servers to fix some residual max_user_connections reached:
mysql55-224.perso.ha.ovh.net
mysql55-142.perso.ha.ovh.net
mysql55-41.perso.ha.ovh.net
mysql55-205.perso.ha.ovh.net
mysql55-75.bdb.ha.ovh.net
mysql55-108.perso.ha.ovh.net
Task is closed.
Date: 2018-12-08 15:01:22 UTC
Saturday 10:10
Below the full list of servers with users reaching max_user_connections:
mysql55-110.perso.ha.ovh.net: 50
mysql55-34.perso.ha.ovh.net: 56
mysql55-196.perso.ha.ovh.net: 3
mysql55-40.perso.ha.ovh.net: 5
mysql55-211.perso.ha.ovh.net: 4
mysql55-58.perso.ha.ovh.net: 82
mysql55-239.bdb.ha.ovh.net: 21
mysql55-237.perso.ha.ovh.net: 7
mysql55-157.perso.ha.ovh.net: 41
mysql55-116.business.ha.ovh.net: 23
mysql55-165.perso.ha.ovh.net: 37
mysql55-167.perso.ha.ovh.net: 216
mysql55-253.perso.ha.ovh.net: 61
mysql55-171.perso.ha.ovh.net: 39
mysql55-127.perso.ha.ovh.net: 61
mysql55-177.perso.ha.ovh.net: 25
mysql55-14.bdb.ha.ovh.net: 4
mysql55-55.perso.ha.ovh.net: 4
mysql55-15.perso.ha.ovh.net: 174
mysql55-106.perso.ha.ovh.net: 117
mysql55-162.pro.ha.ovh.net: 17
mysql55-255.perso.ha.ovh.net: 127
mysql55-121.perso.ha.ovh.net: 126
mysql55-112.perso.ha.ovh.net: 19
mysql55-128.perso: ?
The appropriate patch has been sucessfully deployed on these servers.
Saturday 12:05:
No more users reaching max_user_connections on theirs DB.
To limit unwanted side effects, the patch will be deployed on all servers from Monday.
Date: 2018-12-07 15:03:24 UTC
Those databases errors widely result in 500 errors on your website.
The deployment of the ProxySQL (see http://travaux.ovh.net/?do=details&id=35499) and a tiny error on few customers resulted in a snow-ball effect. 2 causes resulting in a problem, each of them, independently, are not a problem at all. That's why the diagnose was hard, and that's why the change passed our tests.
---
Ces erreurs de bases de données aboutissent généralement à une erreur 500 sur votre site web.
Le déploiement des ProxySQL (cf. http://travaux.ovh.net/?do=details&id=35499) ainsi qu'une petite erreur sur quelques clients ont entraîné un effet boule de neige. 2 causes qui aboutissent à un probleme, mais chacune d'elles, prises indépendamment, n'en provoquent pas. C'est pourquoi le problème était dur à diagnostiquer, c'est également pourquoi le changement a passé tous nos tests.
This incident affected: Web Hosting || Datacenter GRA (Cluster002, Cluster003, Cluster005, Cluster006, Cluster007, Cluster010, Cluster011, Cluster012, Cluster013, Cluster014, Cluster015, Cluster017, Cluster020, Cluster021, Cluster023, Cluster024, Cluster025, Cluster026, Cluster027, Cluster028, Cluster029, Cluster030, Cluster031).