Web Wiz Network Status Announcements

This morning at 07:30 BST, one disk for the shared hosting mail services became corrupted, causing configuration files for mail domains to not load. This corruption was also replicated in redundant hardware, which prevented us from transferring services to redundant backup mail services. 

To restore mail services, we must restore from a backup taken at 20:30 BST yesterday evening. During the restoration period, you cannot send or view emails. Emails sent to you during this time will be delivered to backup delivery mail servers to be delivered to your mailboxes once services come back online.

Recovery may take a few hours, and we hope to have all services optional by 10:30 BST.

Update  7th July 10:30 BST

All mail services are fully restored, although they are running a little slow due to higher demand as emails catch up. It may take a couple of hours for emails received by redundant backup mail services to be transferred to the primary mail service.

Update 7th July 16:30 BST

We are aware of instability issues, with the mail services and are working to resolve the issues.

Update 7th July 17:15 BST

Mail services have been reinstalled and are now back online. We are continuing to see twenty-times the traffic and resources usage with mail services than we would normally expect to see. We will continue to closely monitor mail services.

Update 01:30 8th July BST

We are seeing instability issues with shared hosting mail servers and have engineers working to resolve the issue. 

Update 8th July 05:20 BST

Engineers are restoring 46,000 settings files for mail domains and mailboxes from backup. While these are being restored you will be unable to send or view emails.  The restoration is expected to take 20 minutes. Emails sent to you during this time will be delivered to backup delivery mail servers to be delivered to your mailboxes once services come back online.

Update 8th July 06:00 BST

The restoration process is completed and the mail service are running stable, but CPU usage is still much higher than our engineers would expect. We will continue to monitor and investigate.

Update 8th July 10:30 BST

We have see a couple of brief outages lasting a two to three minutes over the last hour. Our engineers are continuing to investigate. 

Update 8th July 14:45 BST

Mail services have been running without incident for the past four hours. A CPU race condition caused by an indexing issue has also been resolved and CPU usage is now at expected levels. We will closely monitor mail service for the rest of today in case of any further issues.

We want to apologise for any inconvenience caused.

Start of work - At-Risk period:

Start Date: 27th May 2023 Start Time: 16:00 BST

End of work - At-Risk period:

Risk End Date: 28th May 2023 Start Time: 08:00 BST

Service Impact:

We do not anticipate services will be affected during this time. Traffic to our Data Centre is routed using three tier 1 transit providers, and traffic will be carried by our other transit providers during the period allowing us to retain redundancy throughout. 

Reason:

Software upgrading due to service quality sustainability.

Start of work - At-Risk period:

Start Date: 25th May 2023 Start Time: 16:00 BST

End of work - At-Risk period:

Risk End Date: 26th May 2023 Start Time: 08:00 BST

Service Impact:

We do not anticipate services will be affected during this time. Traffic to our Data Centre is routed using three tier 1 transit providers, and traffic will be carried by our other transit providers during the period allowing us to retain redundancy throughout. 

Reason:

Software upgrading due to service quality sustainability.