Postmortem: 9/30 Incident Summary

Incident: Total Due not showing line items total, displaying $0 instead which prevented merchants from completing payments.

Priority: P3

Affected Services:

• Action-Packages: Line-Items, Payment, Payment-Amount-Selection
• Business-Process-Service

Executive Summary

A change was implemented to support partial payments for line-items and payment-amount-selection. This inadvertently caused an issue where the total payment amount would be set to $0 even when the line items were of a higher value, disabling the payment functionality on the UI. As a result, merchants were unable to process payments when using line-items.

Impact

The incident caused disruptions for merchants relying on the “line-items” feature, preventing them from processing payments. Affected customers opened multiple support tickets regarding their inability to process payments, all of which were resolved promptly.

Detection and Response

• Detection: The issue was first reported by a customer at 17:30 UTC, followed by a second report at 18:08 UTC. The team acted quickly by declaring an emergency at 18:10 UTC and setting up a discussion to address the issue.
• Response: The issue was recreated by 18:30 UTC, and a fix was identified. The fix was successfully deployed to production by 20:00 UTC, and monitoring confirmed the resolution of the issue with no recurrence.

Blameless Root Cause

The payment action amount was unexpectedly allowed to be set to $0, which was not accounted for in the design or testing. Insufficient test coverage contributed to the issue being missed before the release.

_______________________________________________________________________________________________________________________________________________

Actions

After a full review of this incident the Authvia development team has introduced new testing parameters to capture the issues involved in this incident.