Designing a Distributed Deployment Architecture for Omnichannel Fulfillment Across 150 Stores

In our previous deployment architecture case studies, we examined how structured discovery and capacity planning guided infrastructure decisions for operations of very different scale. The first, a B2B industrial equipment manufacturer operating at 300 orders per day, employed a monolithic architecture, a unified application server handling both user-facing operations and background processing, because batch jobs ran during dedicated off-peak windows. The second, a B2C fulfillment operation processing up to 11,500+ orders per day at peak, required a distributed architecture separating user-facing workloads from batch processing, because those jobs ran alongside user operations throughout the business day.

That progression illustrates something we see repeatedly: as an operation grows, the infrastructure supporting it does not simply get larger; it changes shape. One machine becomes several, and workloads that once shared a server move onto dedicated ones.

This document presents the next case along that curve: an omnichannel retailer fulfilling orders from 150 locations, a retail store network operating alongside distribution centers, with a catalog of 650,000 stock keeping units and order volume that varies several-fold between normal trading and peak season. Orders arrive continuously from an e-commerce storefront while store associates fulfill them throughout the trading day, and scheduled processing runs against the same system in parallel.

The architecture designed for that profile is documented here: the load analysis behind it, how compute is allocated across user-facing and background workloads, and the sizing rationale for each machine.

The architecture decisions documented here follow the same structured methodology detailed in our foundational engineering blogs:

• How Our DevOps Team Designs Scalable and Secure Cloud Infrastructure for Apache OFBiz – outlines the six-stage process: discovery, analysis and system design, load testing, security, monitoring, and CI/CD planning.
• How the Right DevOps Discovery Questions Shape Apache OFBiz Architecture – explains how discovery questions inform infrastructure decisions around security, availability, scalability, and cost.

This case study demonstrates how business model differences (omnichannel retail versus single-channel fulfillment, distributed store-based fulfillment versus centralized warehousing, continuous order flow versus defined batch windows) translate into distinct deployment architecture patterns. While the discovery methodology remains consistent, the infrastructure design diverges based on actual operational load patterns, user concurrency, and processing requirements.

1. Client Overview

The client operates a high-volume omnichannel retail business delivering products through both an e-commerce storefront and a physical retail network, where stores serve as fulfillment locations alongside dedicated distribution centers. Operations span 150 fulfillment locations with an extensive product catalog, predominantly single-item orders, and significant seasonal demand variation.

1.1 Business Model and Operations

1.2 Product and Inventory Model

The business manages an extensive product catalog distributed across every fulfillment location, requiring inventory accuracy to be maintained network-wide rather than reconciled centrally. Because order brokering can select any store or distribution center as the fulfilling location, a stale availability position at any single node results in either an oversell or a wrongly rejected order. This network-wide inventory visibility is essential to both fulfillment reliability and customer experience.

Orders are predominantly single-item, so order volume and item volume track closely through the system. During peak seasons the operation sees demand rise several times above normal trading levels, and orders received in those periods are fulfilled progressively over subsequent days rather than clearing the same day. Daily fulfillment capacity and daily order intake are therefore distinct measures.

1.3 Sales and Ordering Channels

1. Customer places an order on the e-commerce storefront.
2. Storefront transmits to the data hub via webhook.
3. Data hub relays to Apache OFBiz via the messaging and file transfer layer; managed queue service handles dead letters, object storage carries bulk feeds.
4. Apache OFBiz imports orders every 15 minutes and submits them to brokering.
5. Brokering selects a fulfilling node from the 150 available; store or fulfillment centre staff execute pick, pack and ship.

Apache OFBiz is the system of record for order state, inventory position and fulfillment status across all nodes.

2. Infrastructure Requirements

Designing the deployment architecture began with understanding the expected load and usage patterns. Senior architecture designers and developers from the HotWax team worked through a structured set of discovery questions with the client, covering transaction volume, user concurrency, batch processing, integrations, availability and security.

The key questions and the finalized requirements are summarized below. The methodology behind these questions is covered in detail here.

3. Deployment Architecture Design and Load Analysis

Order Processing Flow

• Order Entry – Orders arrive from the e-commerce storefront to the data hub, then imported to Apache OFBiz every 15 minutes.
• Order Brokering – Apache OFBiz determines which of the 150 fulfillment locations will fulfill the order based on inventory availability and location.
• Store Fulfillment – Assigned orders appear at the selected store or distribution center for staff to action.
• Pick-Pack-Ship – Staff pick items, pack packages, and ship orders with carrier integration.
• Re-Brokering – Orders rejected by the assigned location are returned to brokering and reassigned every 30 minutes.
• Status Synchronization – Fulfillment statuses flow back to the data hub and e-commerce storefront on a scheduled cycle.

Background Jobs Supporting the Flow

• Order import from the e-commerce storefront every 15 minutes
• Order brokering and re-brokering to determine fulfillment location assignment
• Product and catalog synchronization to and from the e-commerce storefront
• Inventory synchronization across all 150 fulfillment locations
• Returns import and processing
• Pre-order and back-order identification and catalog feeds
• Financial, procurement and reporting feed generation
• Fulfillment feed generation and polling between systems

Together, these processes ensure that orders move from entry to fulfillment across a distributed store network while keeping all connected systems updated with accurate inventory and status information.

3.1 Load Calculation and Capacity Planning

To design deployment architecture that performs reliably under real production conditions, we first analyzed how orders flow through the system across different operational patterns.

3.1.1 Order Volume and Seasonal Patterns

The operation experiences significant seasonal variation, with major holiday periods driving peak demand:

Daily Order Overview

What this means:

• Orders are predominantly single-item, so order count and item count track closely through the system
• 180 users operate across 150 fulfillment locations simultaneously
• Orders arrive continuously from the e-commerce storefront, outside store trading hours as well as during them
• Peak-period orders are fulfilled progressively over subsequent days rather than clearing the same day, so daily fulfillment capacity and daily order intake are distinct measures

Order volume can also be derived from the measured request rate, which is the firmer of the two inputs:

• Requests per order: 20 API calls to complete the fulfillment process
• Normal day: 600–900 requests/min ÷ 20 = 30–45 orders/minute
• Observed peak minute: 720 ÷ 20 = 36 orders/minute
• Peak season: 2,400–2,700 requests/min ÷ 20 = 120–135 orders/minute

3.2 Load Calculation and System Capacity Planning

Once order volume and items per order are established, we move to how those orders generate system load across different operational areas.

3.2.1 User-Facing Request Load

User actions create API requests as store associates and business users interact with the fulfillment applications and the commerce platform.

Store and Fulfillment Centre User Load:

• 150 store and fulfillment centre users operating pick, pack, ship and store fulfillment workflows across 150 locations
• 20 API calls per order to complete the fulfillment process
• Peak season: 2,700 requests/minute ÷ 20 calls = 135 orders/minute processed

Business User Load:

• 30 business users managing orders, inventory, catalog and system configuration

Internet-Facing Traffic:

• 2–3 requests/second accepted from e-commerce storefront webhooks and external integrations
• 2–3 × 60 = 120–180 requests/minute, or 5–7% of total system load

Total User-Facing Requests:

• Normal day: 10–15 requests/second × 60 = 600–900 requests/minute
• Peak season: 40–45 requests/second × 60 = 2,400–2,700 requests/minute
• Highest recorded minute from monitoring: 700–720 requests

Expressed per user across the 180-user population:

• Observed peak: 720 ÷ 180 users = 4.0 requests/minute per user
• Peak season: 2,700 ÷ 180 users = 15.0 requests/minute per user

Peak season represents 3.3–3.8 times the observed normal peak (2,700 ÷ 720), and this is the figure infrastructure is sized against. Averaging request counts across the full day is not appropriate for this workload: load is concentrated rather than uniform, and a daily average produces a figure too low to design against.

3.2.2 Background Job Execution

Background jobs are measured by execution time (how long they run) and frequency (how often they run), not requests per minute. The deployment runs 47 scheduled jobs producing approximately 1,370 executions per day.

• Runs per Day = Sum of (jobs at each frequency × runs per day at that frequency)
• Daily Total Execution Time = Runs per Day × Execution Time per Run

The five fulfillment feed jobs each run a generation leg on the integration services machine and a transfer leg on the data pipeline machine, staggered by 5–10 minutes so the two do not collide. The highest frequency anywhere in the deployment is every 15 minutes.

Daily Total Execution Time Calculation:

Execution time per run is not instrumented, so daily totals are expressed against the formula. Applied to the 914 runs carried by the asynchronous tier:

• 914 × 0.5 min = 457 minutes = 7.6 hours/day
• 914 × 1.0 min = 914 minutes = 15.2 hours/day
• 914 × 1.5 min = 1,371 minutes = 22.9 hours/day
• One machine's daily capacity is exceeded at 24 × 60 / 914 = 1.58 minutes average per run

At approximately 1 minute per run, the expected order of magnitude for feed-driven jobs of this type, daily batch execution lands at 15.2 hours, within the 13–17.9 hours measured for the B2C warehouse operation. Beyond 1.58 minutes average, the work exceeds what a single machine can complete in a day, which is what makes the two-machine asynchronous tier necessary.

Job Overlap Calculation:

What sizes an asynchronous machine is not total volume but simultaneous execution:

• 20 jobs are scheduled to fire in the same minute at the top of most hours
• Observed behaviour: 3–4 concurrent jobs on a 4-core machine engage 2–3 cores, or ~0.71 cores per job
• 8 concurrent jobs × 0.71 = 5.7 cores = 36% of the 16-core asynchronous tier
• 16 concurrent jobs × 0.71 = 11.4 cores = 71% of the tier
• 20 concurrent jobs × 0.71 = 14.3 cores = 89% of the tier

Peak Batch Processing Window:

Unlike the B2B manufacturing case study where batch jobs ran during dedicated off-hours (midnight–4 AM), this operation runs batch processing throughout the trading day:

• Order and catalog imports: every 15 minutes, continuously
• Order brokering: 08:00, 11:30, 13:30 and 21:00, with re-brokering every 30 minutes between 08:00 and 14:00
• Inventory synchronization: every 30 minutes to hourly, continuously
• Daily feed jobs cluster in a pre-open window (05:45–08:15) and a night window (21:00–00:15)

Batch jobs overlap with store fulfillment operations throughout the trading day, creating resource contention. This overlap is the primary driver for distributed architecture.

3.2.3 Total System Load Summary

Normal Day (10–15 requests/second, 30–45 orders/minute)

Peak Season (40–45 requests/second, 120–135 orders/minute)

Critical Observation: Unlike the B2B manufacturing deployment where batch processing consumed 4.25 hours daily during dedicated off-hours, and closer to the B2C warehouse operation which ran 13–17.9 hours during operational hours, this deployment runs approximately 15.2 hours of batch processing on its asynchronous tier, spread across the full trading day. With 20 jobs scheduled to fire in the same minute at the top of most hours, resource contention with user-facing operations is continuous rather than periodic.

3.3 Peak Load Analysis: Operational Hours and Concurrent Processing

Daily totals don't reveal the real-time load infrastructure must handle. During operational hours, multiple workloads compete for system resources simultaneously.

3.3.1 Load Pattern During Business Hours

Store fulfillment operations run throughout the trading day across 150 locations, while the e-commerce storefront accepts orders continuously, including outside store trading hours. Scheduled processing runs against the same system in parallel, with recurring jobs at 15-minute, 30-minute and hourly intervals, and daily feed jobs clustering in a pre-open window (05:45–08:15) and a night window (21:00–00:15).

Concurrent User Activity:

• Retail store network: 140 locations with store associates actioning store fulfillment and ship-from-store
• Distribution centers: 10 locations with fulfillment staff running pick, pack and ship
• Business users: 30 managing orders, inventory, catalog and configuration

Each active user maintains 2–3 database connections for fulfillment application operations, open pick lists, pack stations and real-time inventory reservations. Combined concurrent activity requires approximately 360–540 database connections during peak hours, with a further 30–50 held by background jobs, giving approximately 390–590 connections in total.

3.3.2 Database Transaction Impact

Apache OFBiz's entity engine translates each user action into multiple database operations. Brokering across 150 candidate fulfillment locations amplifies database load.

Order Processing – a single order involves:

• Order header insert: 1 transaction
• Customer and address resolution: 4–6 queries
• Order item: 1 item = 1 insert
• Inventory reservation and availability check: 12–18 queries
• Brokering logic across 150 candidate locations: 25–33 queries
• Status history tracking: 3–5 inserts
• Payment and financial record: 4–6 transactions

Total: 50–70 database operations per order

Single-item orders reduce the line-item component of this total, but brokering across 150 candidate locations increases the routing component by a comparable amount. The per-order total therefore sits in the same band as the B2C warehouse operation despite a very different order profile.

Fulfillment – pick-pack-ship process for one order:

• Pick list generation and assignment: 8–12 transactions
• Pick confirmation: 6–10 transactions
• Pack operations: 8–14 transactions
• Shipment creation and carrier handoff: 8–12 transactions
• Inventory decrement and network-wide synchronization: 4–6 transactions

Total: 34–54 database operations per order fulfillment

Database operations are derived at 3.2113 operations per request, the ratio measured in the B2C warehouse operation (2,280 operations per minute against 710 requests per minute at peak).

These database transaction patterns drive deployment architecture sizing decisions around database memory allocation, connection pooling, Multi-AZ configuration, and the critical decision to separate batch processing from user-facing operations.

3.4 Distributed Architecture Strategy: Workload Separation

3.4.1 Why Distributed Architecture Is Required

Batch Processing During Operational Hours:

Unlike the B2B manufacturing case study where batch jobs ran during dedicated midnight–4 AM windows, this operation runs resource-intensive batch processing throughout the trading day:

• Order and catalog import: every 15 minutes, continuously (six separate jobs at this frequency)
• Order brokering: 08:00, 11:30, 13:30 and 21:00; evaluates 150 candidate fulfillment locations per order
• Re-brokering: every 30 minutes, 08:00 to 14:00; reassigns orders rejected by the assigned location
• Inventory synchronization: every 30 minutes to hourly; maintains availability across all 150 locations
• Returns import: every 15 minutes, continuously
• Daily feed jobs: clustered in a pre-open window (05:45–08:15) and a night window (21:00–00:15)

Resource Contention Without Separation:

Running batch jobs on the same server as user-facing operations creates these problems:

• CPU Competition: 20 jobs are scheduled to fire in the same minute at the top of most hours, engaging up to 14.3 of 16 available cores (89%). Store associates may experience slow pick confirmations and pack operations.
• Memory Pressure: Order and catalog import batches trigger garbage collection that pauses user requests for 500–2,000ms.
• Database Connection Saturation: Batch jobs consume 30–50 connections during execution. Combined with 360–540 user connections from 180 users, this reaches 390–590, roughly double typical pool limits (200–300 connections).
• Thread Pool Exhaustion: Brokering evaluates 150 candidate locations per order, holding worker threads substantially longer than a two-facility evaluation. Heavy batch processing consumes application threads, queuing user requests and causing timeouts.

Traffic Pattern Analysis:

• User-facing operations: 2,400–2,700 requests/minute during peak season
• Batch processing: 47 jobs, ~1,370 executions daily, approximately 15.2 hours on the asynchronous tier, running concurrently with user operations
• No dedicated off-hours window: the e-commerce storefront accepts orders continuously, including outside store trading hours

3.4.2 Distributed Architecture Components

Each workload class runs on its own machine, and the two heaviest run on a pair. This is a deliberate allocation rather than a consequence of scale. Dedicating separate machines to user-facing and background processing keeps each workload running smoothly without competing for the same cores, memory or connections, and allows each to be tuned and scaled independently for the work it actually does.

Order Management Main Servers (2):

• Handle all user-facing operations: pick, pack, ship, inventory lookups, order status, store fulfillment
• Serve 150 store and fulfillment centre users and 30 business users across 150 locations
• Optimized for low-latency response times (200–400ms target)
• Deployed as a pair behind a load balancer so user traffic is distributed across both machines, giving consistent response times, zone redundancy and zero-downtime deployment
• Isolated from batch processing resource spikes

Asynchronous Processing Servers (2):

• Handle all 47 scheduled background jobs: order import, brokering, re-brokering, inventory synchronization, catalog feeds, returns
• Process large datasets without impacting user operations
• Deployed as a pair so concurrent jobs execute in parallel rather than queuing behind one another, since 20 jobs fire in the same minute at the top of most hours, which two machines absorb comfortably where one would saturate
• Two machines also provide the headroom for batch work to complete within the day: beyond 1.58 minutes average execution per run the workload exceeds what a single machine could process in 24 hours
• Scaled independently based on batch processing volume

Integration Services Application:

• Handles third-party application programming interface traffic and external system integration
• Separated from both user-facing and batch compute because integration endpoints are the most common source of unpredictable load
• Absorbs retry storms following an upstream outage without touching interactive capacity

Apache Solr Search:

• Dedicated search server for a 650,000 stock keeping unit product catalog and inventory lookups
• Serves both main and asynchronous servers
• Isolated to maintain fast search response times (50–150ms)

Without distributed architecture at this volume, a monolithic server would see user request response times degrade to 800–1,500ms during batch windows, database connection pool exhaustion at roughly double the available connections, and no ability to scale user-facing and background workloads independently.

3.5 Proposed Deployment Architecture

Based on load analysis and the distributed architecture strategy, we propose the following production environment. This architecture separates user-facing operations across a pair of main application servers while isolating background processing on dedicated asynchronous servers.

Fig – Proposed Deployment Architecture for OmniChannel fulfilment

Instance sizing is grounded in measured throughput rather than estimation:

• Application capacity: 720 requests/minute measured at 12.5% CPU on 8 vCPU = 720 requests/minute per vCPU at full utilization
• Database capacity: 2,312 operations/minute measured at 13% CPU on 16 vCPU = 1,112 operations/minute per vCPU at full utilization
• Target sustained utilization: 45–50%, leaving headroom below the 70% scaling threshold

3.5.1 Application Servers (AWS EC2)

Server notes:

• Main App Server ×2: Serves 150 store & 30 business users across 150 locations. Target: 200–400ms response. Peak utilization 46.9%, calculated as 2,700 ÷ (8 × 720). Load-balanced pair for zone redundancy.
• Async Processing Server ×2: Handles 42 scheduled jobs, 914 executions/day. Absorbs the 20 simultaneous starts at the top of each hour at 89% worst case. Pair required: batch exceeds one machine's daily capacity beyond 1.58 min/run.
• Integration Services: Isolates external integration from both user and batch compute. Absorbs upstream retry storms without touching interactive capacity.
• Apache Solr: Memory-optimized for index residency. 2.0 GB index, 4.0 GB during full re-index = 25% of RAM. Fixed rather than burstable performance, since steady query load exhausts burst credits.
• Data Pipeline / SFTP: Burstable instance appropriate for intermittent transfer load. Storage sized for feed retention and buffering.
• Monitoring: Metric collection and alert evaluation for the deployment.
• Jenkins CI/CD: Burstable instance for CI/CD pipelines. Runs occasionally during release cycles.

Why the Main Tier Runs as a Pair: Peak season requires 7.5 vCPU at a 50% target, calculated as 2,700 ÷ (720 × 0.5), against the 8 provisioned across two machines. A pair distributes that load evenly at 46.9% each, absorbs the loss of one machine outside peak season at 25%, and allows rolling deployment with no maintenance window.

3.5.2 Database Configuration (Amazon RDS for MySQL)

Database notes:

• Primary Database: 32 GB enables ~22 GB InnoDB buffer pool caching the full hot dataset. Multi-AZ: <60s failover. 4,335 ops/min at peak after read offload. CPU: 48.8% peak, 13% normal.
• Read Replica: Carries 50% of read operations as a capacity component, not standby. Mirrors primary specs for failover consistency. Lag typically <5s.

Why 8 vCPUs Are Necessary: Peak load generates 8,670 database operations per minute across order processing (50–70 ops/order), fulfillment (34–54 ops/order), brokering across 150 candidate locations, 1,370 daily background job executions, and concurrent sessions from 180 users. With 50% of reads served by the replica, the primary carries 4,335 operations per minute, or 48.8% of an 8 vCPU instance at the measured 1,112 operations per minute per vCPU.

Storage Provisioning: Database storage is provisioned conservatively with auto-expansion enabled rather than allocated generously up front. Relational database volumes cannot be reduced in place; shrinking one requires taking a backup and rebuilding the instance from it, so initial provisioning is an irreversible commitment made on the least reliable information available.

Backup and Recovery Policy:

• Automated daily backups: scheduled within the lowest-traffic window
• Retention period: 7 days for point-in-time recovery
• Encryption: at rest (AWS KMS) and in transit (TLS 1.2+)
• Multi-AZ replication: <60 second Recovery Point Objective across two availability zones
• Cross-region replication: backups held in a second United States region for disaster recovery
• Read replica: enables backup testing without impacting production
• Transaction log backups: continuous for sub-minute granularity recovery

3.6 Network and Security Architecture

All infrastructure resides within a single AWS VPC in the primary United States region, distributed across two availability zones. The main application servers accept HTTPS traffic (port 443) from store associates via a load balancer, while asynchronous processing servers, integration services and Apache Solr remain in private subnets accessible only within the VPC. The database primary and read replica sit in private subnets with no outbound internet access.

Key Security Controls:

• Encryption: All data encrypted at rest (AWS KMS) and in transit (TLS 1.2+)
• Perimeter: AWS WAF inspection and DDoS protection at the load balancer, with TLS termination
• Database Access: RDS accessible only from named application-layer instances via security groups
• Endpoint Restriction: IP allow-listing applied to administrative and integration endpoints
• SSH Access: Key-based authentication only via bastion host, 90-day key rotation policy
• Cross-Region: Encrypted backup replication to a second United States region; no live application traffic

The network design prioritizes security without sacrificing operational efficiency: store associates across 150 locations access systems securely via HTTPS while internal components communicate over the AWS private network backbone.

3.7 Monitoring and Observability

AWS CloudWatch tracks infrastructure health and application performance. Key metrics include CPU utilization (alert at >70% sustained), memory usage (alert at >85%), database connection count (alert at >480), and replication lag (alert at >30 seconds).

Application-level monitoring tracks order import success rates, brokering completion times, re-brokering volumes, inventory synchronization lag across the 150 locations, and store associate response times. Because 20 jobs are scheduled to fire in the same minute at the top of most hours, batch job execution duration is monitored as a production metric rather than an operational detail, because a job that overruns its scheduling interval queues against its own next run.

4. Conclusion

The distributed deployment architecture designed for this omnichannel retail operation demonstrates how requirements fundamentally shape infrastructure decisions. Unlike the B2B manufacturing case study where monolithic architecture efficiently served batch-processing-during-off-hours patterns, and extending beyond the B2C warehouse operation where batch processing overlapped a 9-hour operational window, this deployment separates workloads across paired machines because 47 scheduled jobs run continuously throughout the trading day while an e-commerce storefront accepts orders around the clock.

Notably, order volume alone does not determine architecture. This deployment fulfills from 150 locations against a catalog 217 times larger than the B2C warehouse operation, yet peak database throughput lands within one and a half percent of it: 2,312 operations per minute against 2,280. What drives the architecture is the shape of the workload: how many jobs run, when they run, and whether they overlap with the people depending on the system at the same moment.

This case study reinforces the core methodology: structured discovery translates operational patterns into infrastructure decisions. The same discovery process produces different architectures: monolithic with off-hours batch processing for B2B manufacturing, distributed with a single machine per workload for B2C warehouse fulfillment, and paired machines per workload class for omnichannel retail fulfilling from 150 locations, because the underlying business models and operational patterns differ fundamentally.