Summary and Root Cause:
At 6:41 MST on 7/9/21, Lender Toolkit became aware of an incident which impacted PowerTools subscribers. One of Lender Toolkit's licensing servers failed due to a power outage. The outage caused all PowerTools applications to repeatedly retry pinging the unavailable server.. The number of licensing validation requests exponentially stacked on top of each other, causing the main PowerTools server to overload and shut down. This in turn caused Encompass to throw an error because it could not communicate with PowerTools.
Business Impact:
PowerTools subscribers were presented with a popup error message when opening and closing loan files. Subscribers were advised to save the loan file prior to Closing and click the "Ignore" button on the error popup. If these steps were followed no subsequent impacts were experienced.
LTK Response and Resolution:
Lender Toolkit was alerted to an offline server at 6:41AM MST on 7/9/21, shortly after, the company had assembled to troubleshoot and fix the problem. Once LTK discovered a licensing server was offline, a server admin was sent to bring it back online. Lender Toolkit then terminated communications with the problematic licensing server and increased the number of available workers on the main PowerTools server to handle more licensing requests. By 1:00 PM MST, licensing came back online. Multiple communications were sent to PowerTool subscribers while the incident was being identified, investigated, and resolved. Users were asked to save prior to closing the loan, and then to ignore the application error message to avoid any loss of data.
Remediation:
Lender Toolkit is implementing preventative measures for the future to ensure PowerTools outages do not affect your loan workflow. The team will be adding API throttling and load balancing to PowerTools this week to allow for traffic spikes. We will also be analyzing the code behind PowerTools to ensure any errors that may occur going forward fail gracefully and not disturb daily operations. Within the next 1-2 months, Lender Toolkit will have fully restructured their licensing algorithm so applications will not need to ping the servers as often.