Dollar Amounts changing to $0 in a limited set of business processes
Incident Report for AuthVia
Postmortem

Postmortem: 9/30 Incident Summary

Incident: Total Due not showing line items total, displaying $0 instead which prevented merchants from completing payments.

Priority: P3

Affected Services:

  • Action-Packages: Line-Items, Payment, Payment-Amount-Selection
  • Business-Process-Service

Executive Summary

A change was implemented to support partial payments for line-items and payment-amount-selection. This inadvertently caused an issue where the total payment amount would be set to $0 even when the line items were of a higher value, disabling the payment functionality on the UI. As a result, merchants were unable to process payments when using line-items.

Impact

The incident caused disruptions for merchants relying on the “line-items” feature, preventing them from processing payments. Affected customers opened multiple support tickets regarding their inability to process payments, all of which were resolved promptly.

Detection and Response

  • Detection: The issue was first reported by a customer at 17:30 UTC, followed by a second report at 18:08 UTC. The team acted quickly by declaring an emergency at 18:10 UTC and setting up a discussion to address the issue.
  • Response: The issue was recreated by 18:30 UTC, and a fix was identified. The fix was successfully deployed to production by 20:00 UTC, and monitoring confirmed the resolution of the issue with no recurrence.

Blameless Root Cause

The payment action amount was unexpectedly allowed to be set to $0, which was not accounted for in the design or testing. Insufficient test coverage contributed to the issue being missed before the release.

_______________________________________________________________________________________________________________________________________________

Actions

After a full review of this incident the Authvia development team has introduced new testing parameters to capture the issues involved in this incident.

Posted Oct 04, 2024 - 09:08 PDT

Resolved
This incident has been resolved.
Posted Sep 30, 2024 - 13:29 PDT
Monitoring
We have released a fix that has resolved this issue. We will continue to monitor
Posted Sep 30, 2024 - 13:05 PDT
Identified
When using Line-items it is not reading the amount and defaulting to $0
Posted Sep 30, 2024 - 11:30 PDT
Investigating
We are currently investigating this issue.
Posted Sep 30, 2024 - 11:16 PDT
This incident affected: api.authvia.com (business-process-service, transaction-service).