Potential upload issues on Learn365 (LMS365)
Incident Report for ZENSAI
Postmortem

Description: On April 29th, internal monitoring system alerts were triggered by errors which potentially could have affected Learn365 (LMS365) users' ability to upload files within the platform.

Root Cause Identification and Analysis: The root cause of the incident was identified as code changes related to the uploading documents action. These changes led to transient crashes in the app service, potentially impacting users' ability to add learning content.

Resolution: To resolve the issue, the problematic section of the code was swiftly fixed, restoring normal functionality for file uploading within the Learn365 platform.

Remediation Items: As a response to this incident, we are taking the following steps to prevent similar issues in the future:

  1. Review of Internal Monitoring and Testing Processes:
* Despite the lack of concrete evidence that customers were affected, we used this incident as an opportunity to review our internal monitoring and testing processes.
* We are enhancing our monitoring system to better detect and respond to similar issues in the future.
* Our testing procedures will also be strengthened to prevent code changes from causing disruptions to critical functionalities such as file uploading.

We apologize for any inconvenience caused and appreciate your patience and understanding during this incident. If you encounter any difficulties with file uploading, please do not hesitate to contact us.

Posted May 10, 2024 - 12:26 CEST

Resolved
We can confirm that the issue with the files uploading is now resolved.

If you notice any issues going forward, please reach out to us.
Posted Apr 30, 2024 - 12:41 CEST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Apr 30, 2024 - 10:33 CEST
Identified
We have identified the cause of the issue and have prepared a hotfix to address it. This hotfix, scheduled for release on April 30th, will eliminate the possibility of errors occurring during file uploads in the affected regions.

Additionally, we would like to inform you that the Learn365 (LMS365) April release is currently being deployed in the Japan and Australia regions. This release also includes work items aimed at resolving the issues that led to both incidents.

We will continue to keep you updated as we gather more information.
Posted Apr 29, 2024 - 16:07 CEST
Investigating
We are currently investigating internal monitoring system alerts regarding errors that may potentially affect Learn365 (LMS365) users' ability to upload files within the platform.

We are working diligently to minimize any impact these errors may cause and are striving to resolve them as quickly as possible. Should you encounter any difficulties with file uploading, please do not hesitate to contact us.

We will provide further updates as soon as the cause of the issue has been identified.

Learn365 (LMS365) April release in Japan and Australia regions is postponed until further notice.
Posted Apr 29, 2024 - 13:50 CEST
This incident affected: LMS365 North Europe, LMS365 United States, LMS365 Canada, LMS365 United Kingdom, LMS365 Germany, LMS365 Switzerland, LMS365 Government Community Cloud, LMS365 Government Community Cloud High, and LMS365 for Microsoft Teams.