Customers may observe severe performance degradation and errors (IM-39794)
Incident Report for Docusign
Postmortem

RCA for eSignature North America Service Disruption on April 30, 2024

Impact Summary:

Customers on DocuSign’s NA2 site may have experienced processing request failures or high latencies during the affected time period.

Cause:

Abnormal and extreme sub-function processing volumes affecting the handling of metadata associated with sending and signing activity caused service degradation.

Resolution:

To address the degradation, prompt reactive action was taken on April 30 to (1) suspend Connect and Bulk Send capabilities, (2) enable reactive throttling of certain non-core system processing activities, and (3) disable status updates to reporting and search functionalities.  Upon system recovery, Connect and Bulk Send were promptly re-started and reactive throttling was also disabled.

Between April 30 and May 2, proactive action was taken to deploy a permanent fix as well as remediate affected status updates with reporting and search.  Reporting and search functionalities also resumed normal operation on May 2.

Next Steps:

This issue is mitigated.  We are nonetheless undertaking ongoing efforts to detect potentially abnormal processing activity that may occur otherwise in the future and to proactively apply remediation steps as necessary.

Posted May 04, 2024 - 18:20 UTC

Resolved
System performance maintains healthy. Please follow the status page https://status.docusign.com/incidents/7b5dlw66lvt3 for updates regarding the delayed accuracy of envelope states in dashboards and search results which customers will continue to face.
Posted Apr 30, 2024 - 17:50 UTC
Update
We are continuing to monitor for any further issues.
Posted Apr 30, 2024 - 16:40 UTC
Monitoring
The team is continuing to monitor the recovery of performance. We have re-enabled flows including Connect & Bulk send messages, however customers should expect to continue to observe delayed envelope states represented in dashboards and search results.
Posted Apr 30, 2024 - 16:25 UTC
Update
To alleviate the observed influx in load, customers may observe delayed accuracy of envelope states represented in dashboards and search results.
Posted Apr 30, 2024 - 16:17 UTC
Update
We continue to investigate the incident. To help mitigate faster we've disabled flows that are not part of the core Sending & Signing experience. Connect & Bulk send messages will be delayed for now. We continue to investigate other actions to recover.
Posted Apr 30, 2024 - 15:49 UTC
Update
We are continuing to take steps toward recovery.
Posted Apr 30, 2024 - 15:35 UTC
Identified
We are taking steps towards recovery.
Posted Apr 30, 2024 - 15:18 UTC
Investigating
We are currently investigating this issue.
Posted Apr 30, 2024 - 14:50 UTC
This incident affected: eSignature (NA2).