LogoLogo
Become a PartnerStatus Page
  • get started
    • Welcome to 360Dialog
    • Quickstarts
      • Register as a Meta Tech Provider
      • Add a WhatsApp Number
      • Set up Integrated Onboarding
      • Create a Message Template
      • Send a Message
      • (temp title) Getting started as a partner
    • Pricing & Billing
      • Payment Methods for BR: Pix & Boleto
      • Month Closing Invoice (MCI)
    • Get API Access
    • Tech Provider Program
      • Understanding the Meta Tech Provider Program
      • Becoming a Meta Tech Provider: A Step-by-Step Guide
  • MM Lite Beta Program
    • Onboarding
    • Sending Messages
    • Expected Errors & Programming Languages
    • MM Lite Sandbox Testing
      • Sending Messages using MM Lite Sandbox
      • Expected Erros & Programming Languages
  • Partner & Account Management
    • Solutions for Partners
      • Benefits for Partners
      • Partner Hub
      • WABA Management
    • Account Setup & Management
      • Prerequisites and best practices for account creation
      • Account Structure
      • Account Creation
  • Integrations & API Development
    • Webhook Events & Setup
      • Webhook Events (Partner & Messaging API)
      • Webhook Configuration & Management
    • Integration Best Practices
      • Architecture and Security
      • Best Practices for Designing Integrations
        • Design a Stable Webhook Receiving Endpoint
      • Sizing Your Environment Based on Expected Throughput
      • Design a Resilient Message Sending Service
      • Integrated Onboarding
        • Basic Integrated Onboarding
        • Custom Integrated Onboarding
        • Host Your Own Embedded Signup
      • API Key Authentication for the Partner API
  • API Reference
    • Messaging API
    • Messaging Health Status
    • Sandbox (Test API Key)
    • Differences Between Cloud API and On-Premise API for Partners
      • [will be deprecated] Messaging with On-Premise API
    • Partner API
  • WABA Management
    • Managing WABA Accounts
      • Using the Partner Hub to manage Clients and Channels
      • Using the Partner API to manage Clients and Channels
      • Partner Permission to Generate API Key
      • Checking Usage & Statistics
    • Phone Number & Hosting
      • Migrating a Phone Number
        • Migrate number from Meta or alternate BSP to 360Dialog
        • Migrate a phone number to a new WABA
        • Migrate a phone number to a new WABA to change messaging Currency
      • WhatsApp Coexistence
        • Coexistence Onboarding
        • Coexistence Webhooks
      • Hosting type Change (On-premise API to Cloud API)
    • WABA Profile & Compliance
      • Display Name Guidelines
      • WABA Profile Info
      • WABA Policy Enforcement
      • Business Account Verification
    • Partner change (between 360Dialog Partners)
    • WhatsApp Flows
    • Accelerated Onboarding
    • Business Account Verification
  • Messaging
    • Sending & Receiving Messages
      • Conversations
      • Receiving messages
      • Before sending a message
    • Message Types
      • Conversational Components
      • Checklist for Message Broadcasts and Campaigns
      • Text messages
      • Interactive messages
        • Single and Multi Product Messages
        • Location Request Messages
        • Address Messages (India and Singapore only)
      • Contacts & Location Messages
    • Template Messages
      • Template Library
      • Template Elements
      • Sending Template Messages
      • Authentication Templates
        • Zero-Tap Authentication Templates
        • One-Tap Autofill Authentication Templates
        • Copy Code Authentication Templates
      • Catalog Templates
      • Product Card Carousel Templates
      • Single-Product Message Templates
      • Coupon Code Templates
      • Limited-Time Offer Templates
      • Multi-Product Templates
    • Media Messages
      • Upload, retrieve or delete media
    • Messaging Health & Troubleshooting
      • Messaging Limits & Quality Rating
      • Errors While Messaging
      • Messages statuses
  • Commerce & Payments
    • Payments (India Only)
      • Receive WhatsApp Payments via Payments Gateway
      • Receive WhatsApp Payments via Payment Links
      • Order Details Template Message
    • Payments (Singapore Only)
      • Receive WhatsApp Payments via Stripe
    • Products and Catalogs
      • India Businesses: Compliance for Commerce
  • Support & Updates
    • Help & Troubleshooting
      • FAQ
      • Error Code Reference
      • User Support Documentation
        • Tutorials
          • Page How to Troubleshoot Login Issues in the 360dialog Hub
          • How to Report an Issue
          • How to Contact Support
        • How-To Guides
          • Raise a support request
          • Requesting a Status Update for Business Verification
        • Reference
          • Troubleshooting Embedded Signup Issues
          • Commerce Policy Checks for WhatsApp API Access
          • Support Availability & SLAs
          • Incident Issue Prioritization
          • Case Types Table
        • Explanation
          • Why Business Verification Takes Time
          • Why Some Issues Are Not Emergencies
          • Understanding the Support Process
      • Support with Meta
    • Imprint & Data Privacy
    • Product Updates & News
      • How to ensure your Template Messages will be received
      • Integrated Onboarding new look and Campaigns Best Practices
Powered by GitBook
On this page
  • Build your Integration with a resiliency to errors: implementing a Message Queue and a Retry mechanism with exponential backoff
  • Different fault types
  • Implementing a comprehensive solution

Was this helpful?

  1. Integrations & API Development
  2. Integration Best Practices

Design a Resilient Message Sending Service

Ensure that messages are guaranteed to be sent, and never lost

PreviousSizing Your Environment Based on Expected ThroughputNextIntegrated Onboarding

Last updated 1 month ago

Was this helpful?

Build your Integration with a resiliency to errors: implementing a Message Queue and a Retry mechanism with exponential backoff

To ensure that clients don't lose any messages or become alarmed by minor glitches in the 360dialog <> partner connection, there are two patterns which if combined, can help us create a more resilient and reliable messaging system: a retry mechanism, and a message queue.

Your hosted WABA instance combines message queues and retry mechanisms to ensure that callbacks are always delivered to your web-hook endpoint, and messages are always delivered to the WhatsApp network. In order to achieve this level of reliability to your systems and integration(s), we strongly encourage you to implement a similar solution.

Let’s break down the problem and analyse it step by step.

Different fault types

Clarity on fault types is essential for Identifying the right solution It's crucial to understand that faults can come in different forms: transient, intermittent, and permanent.

Transient and intermittent faults are time-bound disruptions. Examples include a service glitch that requires a restart, a temporary loss of connectivity between your systems and 360dialog, or scheduled maintenance procedures that result in weekly service restarts.

Permanent faults, on the other hand, persist until the faulty component is fixed. Examples include long-lasting connectivity issues with your ISP and 360dialog, a full-scale malfunction of a cloud provider, and a persistent resource shortage in 360dialog systems that leads to the failure of all API requests for an extended period.

How to solve transient and intermittent faults - Retry Mechanism

For transient and intermittent faults, you can establish a retry mechanism where failed requests are repeatedly attempted. Every time a retry is initiated, the waiting time is increased by a factor of 2, as demonstrated in the diagram below.

Warning: Without an exponential backoff, your integration may send too many requests to 360dialog's systems, triggering rate limiting that can temporarily prevent access to the API (waba.360dialog.io) for your system.

How to solve permanent faults - Message Queue

While a retry mechanism with exponential backoff may be sufficient to handle transient and intermittent faults and/or with low message volumes, it may not be able to deal with permanent faults. Eventually, a retry mechanism based on memory storage will consume all available memory, causing it to fail and potentially lose messages.

A software design pattern that can be implemented to handle such cases is a message queue.

A message queue is an asynchronous service that facilitates the transfer of data between two points. The entity that initiates the message is referred to as the Producer, and the recipient is known as the Consumer.

Message Queue Functionality

The Producer begins the process by adding a message to the message queue, where it remains until it is retrieved by the Consumer. The Consumer retrieves the message(s) and attempts to send it to the 360dialog API. If the message is successfully sent, it is removed from the message queue. In the event of failure, the message is returned to the queue, ensuring that no unsent messages are ever lost. The implementation of this process is demonstrated in the following diagram.

Implementing this solution will guarantee that the messages will always be delivered regardless of the destination status, as the messages will sit in the message queue until the destination is reachable.

Note: Failed messages should only be returned to the Message Queue when the failure is due to a server error (5XX status). Client errors (4XX status) signify that there is something wrong with the message request itself (e.g. sending to the wrong url), so the message will always be rejected by 360dialog. Client errors must be removed from the queue, and handled elsewhere.

Implementing a comprehensive solution

For maximum resilience and reliability, we suggest combining a message queue with a Consumer that implements a retry mechanism with exponential back-off. The message should only be removed from the queue once successful delivery is confirmed. This way, your system will be able to handle its own malfunctions.

Implementation examples and suggestions

To see examples of how to implement a retry mechanism with exponential back-off, we recommend checking out the following resources:

When implementing a message queue, it's important to note that it doesn't have to be built using commonly-used tools like Celery and Redis, RabbitMQ, or Kafka. For example, the widely used mail server Postfix implements a file-based approach that leverages the atomic file rename and move characteristics of UNIX file systems.

You could also implement a message queue on a SQL database, similarly to how it is implemented in the WABA instance itself.

Pseudocode Implementation of a Message Sending Queue, with a Built-in Retry Mechanism

The first step of sending a message is to add the message to the message_queue

import message_queue

def send_message(message):
    message_queue.add_message(message, backoff_time=0)

The second step of sending a message is done in another dedicated process: the main function will retrieve messages from the queue, and attempt to send the message. If the message is successfully sent, the message is deleted from the queue, otherwise it is returned to the message queue with an increased backoff time.

import message_queue

BACKOFF_LIMIT = 86400

        
def deliver_message_to_waba_api(message):
    # implementation of message sending via 360dialog API
    pass


def send_message(message, backoff_time):
    if backoff_time:
        time.sleep(backoff)

    is_sent = deliver_message_to_waba_api(message)
    if not is_sent:
        if backoff_time == 0:
            backoff_time = 1

        if backoff_time < BACKOFF_LIMIT:
            # Increment backoff time only if lower than 24H.
            # In some cases it could make sense to remove the message from the queue
            # after a certain time limit.
            backoff_time *= 2

    return is_sent, backoff_time


def main():
    messages = message_queue.get()
    for message, backoff_time in messages:
        is_sent, backoff_time = send_message(message, backoff_time)
        if is_sent:
            message_queue.delete_message(message)
        else:
            message_queue.return_to_queue(message, backoff_time)

implements a queue that can be configured to work on files, sqlite, or MySQL.

If you're working in a cloud environment, you can explore the message queue options available in services like , or .

https://keestalkstech.com/2021/03/python-utility-function-retry-with-exponential-backoff/
https://www.npmjs.com/package/exponential-backoff
https://www.bayanbennett.com/posts/retrying-and-exponential-backoff-with-promises/
This Python library
GCP
AWS
AZURE
diagram of retry mechanism interacting with an intermittent server