04/03/2021 10:00 GMT All Circle B machines are correctly configured [INCIDENT STOPPED]

After the incident, we decided to change the network configuration to use bonding (use both network interfaces connected to the same swicth). All the machines have been rebooted to take into account the new configuration. There is no more DHCP at all in the machines.

03/03/2021 19:30 GMT Circle B machines are online again

All our machines lost their IP address because the DHCP server connected to them was removed. Even though the machines weren't using DHCP address, when the DHCP server stopped answering the DHCP requests from the machines, the DHCP client removed all addresses configured on the machine.

The solution was to replug the DHCP server and attribute IP addresses. Then, we killed the DHCP clients on all the machines.

03/03/2021 16:30 GMT All Circle B machines are not available [INCIDENT STARTED]

Due to an incident inside Circle B datacenter, all our rack is inaccessible.