The cause:
As you were aware, last week we had launched an upgrade to the calendar system 10pm on Sunday evening. Prior to this we were extensively testing the update, and all tests were coming back fine. The launch of the update itself was in bad timing as completely unrelated to this our storage service provider 'Rackspace' faced unexpected downtime.
Rackspace are a giant within the server industry, and although we do not host our core servers with them, our file storage is ran through them. Because of the incident they faced it meant that we had to wait 40 seconds for every file we downloaded.
What we have done:
1) Our team were working around the clock to get things solved. As a temporary fix we began to disable features that required Rackspace, which is why you may of experienced some odd things happening throughout last week.
Examples being:
- The ICS file that is sent with appointment confirmations.
- Email attachments.
- User avatars.
.. and more
By doing this it allowed us to have an almost normal day on Saturday.
2) Although our servers were not showing any sign of 'Load', whilst problem solving what was going on, we used this as an excuse to upgrade them. We also have HEAVILY optimized the calendar code. It is difficult for you to compare the speeds as all the upgrades are all in place. But to put it into perspective, our tests showed that we were able to speed up the loading of contact files greatly (especially those that have a lot of data on them, just give it a go).
3) We are putting things into place to not be so reliant on 3rd parties infrastructures, but sometimes these things can be unavoidable, and in fairness to Rackspace we have had no such issues in the past.
Thank you so much for your continued understanding. Although that it was not a fault at our end, we strive to do better, take responsibility and will work on our processes on keeping you guys in the loop a lot faster. Obviously our support went into meltdown, but we are in the process of getting back to all tickets as soon as possible. We have added Alex to our support team so be sure to give him a big hello.
We have just patched, which has put things back to normal. We will closely be monitoring the performance today, as Rackspace still have not confirmed there systems are officially stable, but all seems well.
You can keep track of this here, which will explain more in technical terms also what has been occuring with them:
https://status.rackspace.com/index/viewincidents?group=15
With love, the Pabau team.
Did this answer your question?