Close

Incident Response Policy

Summary

This plan will outline what steps DataBiz Solutions will take in the event of a catastrophic failure either electronically (hacking, ransomware, 3rd party outages) or physical (severe storms, earthquakes etc.)

Overview of the company infrastructure

  • DataBiz Solutions is a small company and each of its employees works remotely in separate locations and are quite mobile.
  • The company uses AWS, one of the world leaders in providing cloud services, as its primary provider of hard infrastructure and network connectivity.
  • We rely primarily on a single live production server as both our web and database server and all customer data resides on this server which is located in Dublin.
  • We also have a test server on the same infrastructure which is a replica of the live server but without the customer data.
  • Each client has its own separate database on our server which means that an issue with one client's data only affects them and not our whole client base.

Backups

We have the following backup scenarios in place.

  • Encrypted backups of our databases are taken every 3 hours and stored offsite. Individual databases for each client can be restored if required. The Retention Policy is:
    • Daily – 60 days
    • Weekly – 26 weeks
    • Monthly - 24 months
    • Yearly – 4 years
  • Encrypted backups of our webserver file structure is taken once daily and stored offsite.
  • Daily snapshots of the servers are taken each night and stored on AWS. At any stage a new instance (server) can be created from these snapshots. There are 7 daily snapshots available at any one time.
  • Utilising the AWS Elastic Disaster Recovery service means we have our live server continuously replicating to another region, (Paris, France) and can recover a server to one of the following Points in Time:
    • 10 minute intervals within the last hour
    • Hourly intervals within the last 24 hours
    • Daily intervals within the last day for the previous 7 days.
  • All code relating to the web application is stored on a private GitHub repository and can be accessed whenever required. A copy of the code on the server itself is also included in the daily backup of the webserver file structure.

Incident Plan

Option 1 - Catastrophic Failure

Should an incident arise whereby there is a catastrophic failure and the live server is completely unavailable, the following steps will be taken:

  1. The following personnel will meet (either via video call or conference call) to co-ordinate the response.
    • Sean Mulkerrins (company owner)
    • Senior developer
    • Systems Administrator
    • At least of two of the above three will be available at all times and two will be enough to co-ordinate the Disaster Recovery Plan.
    • Regardless of the type of incident, employees work remotely and are mobile so if the issue is causing problems for employees in a specific geographic location it will be possible to move an area where they can get online and work to get the issue resolved.
  2. The first course of action will be to determine whether or not the current server is viable and can be brought back online.
    • If it can, then the time frame required will need to be determined to see how long it will take:
      1. The System administrator and lead developer will determine what the issue is and what needs to be done to get it back online
      2. The necessary fix/update will be applied to bring the server online
      3. The data will be tested to ensure it's integrity
      4. The maximum expected turnaround time to bring the server back online would be 8 hours. If is estimated that it will be longer than that the replica server will be used.
    • If it can't be brought back online within a reasonable amount of time – less than 8 hours - then we will initiate the launching of the replica server located in Paris.
      1. It may be necessary to test multiple versions (Points in Time) of this replica server until determining the most recent point at which the integrity of the data can be verified
      2. Changes to DNS will need to be made
      3. Expected turnaround time to having a live server available should be between 2 to 4 hours once the decision to go with the replica has been made and we can use the data from a certain Point in Time in the replication process
    • Please note that during the recovery of the server, all public access to the server will be disabled until it is fully restored and operational to ensure the integrity of the data available to our clients. During this time it may be necessary to utilize a combination of our various backup scenarios to get the most data possible for our clients.
    • Regardless of the scenario, all clients will be contacted to notify them of the outage and the expected restoration of service timeframe and will be made aware of any updates during the restoration process
  3. If the replica server has been used, the next step will be get the original or replacement server back on line and revert back to the scenario where we have a live server and replication to a secondary server.
  4. Once the server has been brought back online and client access has been restored a review of what caused the issue will be undertaken to determine whether it something that we could have prevented or if the issue was out of our control.

    If it is something that could have been prevented an action plan will be implemented to address the issue and implement the necessary changes to ensure it doesn't happen a again.

Option 2 – Single Customer failure

Should an incident arise whereby a single customer can't access the system the following steps will be taken:

  1. The Support person who takes responsibility for the issue will co-ordinate the response with the developer and systems administrator.
  2. The support person will be responsible for keep the client updated on all that is been done to bring them back online.
  3. The first step is for the developer and systems administrator to determine what the issue is – systems issue, database corruption, coding issue etc..
  4. Once the issue is diagnosed then the necessary remedial steps will be taken to resolve, this may involve some or all of the following:
    • Changes to a server config file
    • Changes to code
    • Restoration of the entire client database or possibly just some tables within it
  5. Regardless of the changes to be made to resolve the issue, the change will be first deployed to the test server to verify that there is no knock-on effect from these change to other clients.
  6. Once satisfied that the changes will not cause any conflict, they will be deployed to the live server. Checks will be made to ensure all is working for the client and they will then be informed that they have their access restored.
  7. Expected turn around time to resolution on these types of scenarios are between 1 to 8 hours.
  8. Once the server has been brought back online and client access has been restored a review of what caused the issue will be undertaken to determine whether it something that we could have prevented or if the issue was out of our control.

If it is something that could have been prevented an action plan will be implemented to address the issue and implement the necessary changes to ensure it doesn't happen a again.

Preventative Measures

Patch Management

  • Security patching on servers is done on a monthly basis, usually the first week of the month following our patch management process which involves testing the updates on the test server first before deploying to the live server.

Vulnerability scanning

  • Vulnerability scans are run on a quarterly basis to highlight any possible vulnerabilities on our system that need to be addressed.

Disaster Recover Testing

  • A random client data base is restored to our test server on a monthly basis to ensure that our database backups are working correctly.
  • We utilize the AWS Elastic Disaster Recovery service to replicate our live server. This service allows us to test the success of the replication by running non-disruptive tests to confirm that implementation is complete. We can perform non-disruptive recovery and failback drills – this is done on a quarterly basis.
  • New instances of the daily server snapshot are created on a quarterly basis to ensure that is working correctly.
  • All of the above processes allow us to familiarise ourselves with the disaster recovery processes so we are ready to implement them if needs be.