User avatar
jsantos
Site Admin
Topic Author
Posts: 4124
Joined: 20 Jan 2012, 17:18
Contact:

Incident Report Hosting Outage

06 Nov 2018, 23:04

On October 29, 2018, our server hosting platform suffered an outage that lasted multiple hours. After they restarted the systems, they discovered that the backup system was corrupted. This is the server where we store our forums, tracker, buildbot and a few more internal subsystem. After a few hours of panic and stress, our administrator started to work hard to recover as much as possible. Fortunately we recovered all the tickets in the tracker and almost all forums content (only 18 topics were lost and we were able to recover the subject text hoping our community helps us to fill the rest of lost information).

We are really sorry about this incident. We are in the middle of an important release, so this happened at the worst time. But, we took this opportunity to improve many things in our infrastructure. For example:
  • Many improvements to our internal bug tracker. As so many times demanded, finally our tickets are public by default to everybody. This will only affect the new ones and of course still there is the option to create a private report. We are also planning to deprecate our Trello board and move everything to the tracker. That way, our internal roadmap will be directly shared to our users.
  • We tried to move the forums to Discourse. But after many days testing we discovered we were not sure about this change so we decided to improve our phpBB board. We have dedicated many hours to polish everything. We hope you like all the new features we implemented like 'tags', 'mark as solved', 'similar topics' and better responsive theme.
  • There is a new sub-forum for official announcements (where you are reading this 🙂 right now).
  • And of course we implemented a backup plan to be no longer dependent of our host.
Again, thanks everybody for being so patient!

Who is online

Users browsing this forum: No registered users and 0 guests