Problem Description & Scope

Affected time range (times are CDT): June 24, 4:00 AM to June 24 10:30 AM

Description: An update to the ClamAV malware database introduced a signature that flagged any zip file as a virus. Therefore, any attempted course import, whether through the API or the UI, of a zipped package failed due to a failed virus scan check.

Root Cause

As part of our safety checks before we import a provided course, we use ClamAV to scan the file for any viruses. ClamAV maintains a database of known malware signatures which is automatically updated on a regular basis. This morning at approximately 4 AM CDT, a new update to this database was pushed out which contained a test signature that flagged any zip files that were scanned.

We began to receive tickets about these import issues from customers using both the UI and the API and getting the same result: a message that their uploaded course failed our virus scan. Therefore we could narrow the issue down to the virus scanner itself, where we then located the signature that was generating all of these failures. We tested this theory by using one of our own sample courses, and it too returned the same error message.

Upon further investigation, we noticed other people using the ClamAV tool posting in GitHub about the very same signature. They too had tested with known good zip files that were still being flagged based on this signature. We therefore concluded that this particular signature was most likely a false positive and not likely to indicate that the flagged file contained malware.

Corrective Action

We added the signature in question to our whitelist and, after verifying that it solved the issue and allowed course imports, pushed it out to our production environment. After confirming with customers that reported import issues that their courses could now be successfully imported, we concluded that this issue was resolved.

Future Action

As part of our corrective action, we notified ClamAV of the potential false positive through their online portal. Once we are notified that the database has been updated and the offending signature has been changed, we can safely remove it from our whitelist.

Timeline

Below is a timeline of events occurring on June 24, 2022 (times in CDT):

• 4:02 AM - We started to see an elevated number of imports that were flagged by our automated virus scan.
• 6:28 AM - An increase in tickets began to flow in as normal business hours started and customers noticed that their imports were failing.
• 7:05 AM - We were provided with our first sample courses that had failed the check, and we started to investigate whether there was something in the course that was causing this error.
• 7:59 AM - At this point, we had received a number of tickets about courses from multiple authoring tools failing. We tested the import with our own sample courses which threw the same error, so we determined that the issue had to lie with ClamAV.
• 8:10 AM - The incident report was opened and published to our status page.
• 8:37 AM - We identified the signature that was causing our sample course to be flagged by ClamAV, and verified that the same signature was causing the sample courses from customers to fail on import as well.
• 8:56 AM - After finding information that other ClamAV users were having issues with this signature, we added it to our whitelist and tested that the scan would still catch and identify other malicious files in the course zip.
• 9:04 AM - A hotfix was built and prepared for release.
• 9:53 AM - The fix was published to our production environment, and we verified that we could import courses again.
• 10:34 AM - After confirmation from our customers that their imports were working as expected again, we marked the issue as resolved.