Designing the Distributed Deployment Architecture for High-Volume B2C Fulfillment

In Part 1 – Understanding Client Requirements and Load Patterns for a High-Volume B2C Warehouse Operation, we covered the client profile, the discovery process, and the load analysis — order volumes, background job schedules, concurrent user patterns, and database transaction load. Now, in this blog, we cover how those findings translate into the actual deployment architecture: the distributed server setup, instance sizing, database configuration, network and security design, monitoring, and scalability planning.

3.4 Distributed Architecture Strategy: Workload Separation

For this deployment handling 11,500 orders per day during peak seasons with batch processing running throughout operational hours, a distributed architecture separating user-facing operations from batch processing is the appropriate infrastructure approach.

3.4.1 Why Distributed Architecture Is Required

Batch Processing During Operational Hours:

Unlike the B2B manufacturing case study where batch jobs ran during dedicated midnight–4 AM windows, this B2C operation runs resource-intensive batch processing throughout the 9-hour operational window:

• Pick waving: 6 AM – 4 PM (10 hours) — processes 450 orders per batch every 20 minutes
• Order routing: Continuous — evaluates facility assignment every 15 minutes
• Replenishment: 5 AM – 5 PM (12 hours) — generates stock movements every 30 minutes
• Order import: 8 AM – 5 PM (9 hours) — polls data hub queue via GCP Pub/Sub every 10 minutes

Resource Contention Without Separation:

Running batch jobs on the same server as user-facing operations creates these problems:

• CPU Competition: Pick waving spikes CPU to 70–85% for 4–10 min every 20 min. Warehouse users may experience slow pick confirmations and pack operations.
• Memory Pressure: Order import batches (up to 1,000 orders) trigger garbage collection that pauses user requests for 500–2,000ms.
• Database Connection Saturation: Batch jobs consume 30–50 connections during execution. Combined with 140–210 user connections, this approaches pool limits (200–300 connections).
• Thread Pool Exhaustion: Heavy batch processing consumes application threads, queuing user requests and causing timeouts.

Traffic Pattern Analysis:

• User-facing operations: ~710 requests/minute during peak hours (10 AM – 2 PM)
• Batch processing: 13–17.9 hours daily, running concurrently with user operations
• No dedicated off-hours window: Operations run 7 days/week during business hours

3.4.2 Distributed Architecture Components

Primary Application Server:

• Handles all user-facing operations: pick, pack, ship, inventory lookups, order status
• Serves 70+ warehouse users and 20+ admin users
• Optimized for low-latency response times (200–400ms target)
• Isolated from batch processing resource spikes

Batch Processing Server:

• Handles all scheduled background jobs: order import via GCP Pub/Sub, routing, pick waving, replenishment
• Processes large datasets without impacting user operations
• Tolerates longer execution times (4–10 minutes for pick waving batches)
• Scaled independently based on batch processing volume

Apache Solr Search:

• Dedicated search server for product catalog and inventory lookups
• Serves both primary and batch servers
• Isolated to maintain fast search response times (50–150ms)

Without distributed architecture at this volume, a monolithic server would see user request response times degrade to 800–1,500ms during batch windows, database connection pool exhaustion, and inability to scale workloads independently.

3.5 Proposed Deployment Architecture

Based on load analysis and the distributed architecture strategy, we propose the following production environment. This architecture separates user-facing operations onto a primary application server while isolating resource-intensive batch processing on a dedicated batch server.

Fig – Proposed Deployment Architecture

3.5.1 Application Servers (AWS EC2)

3.5.2 Database Configuration (Amazon RDS for MySQL)

Why 8 vCPUs Are Necessary: Peak load generates 2,280 database operations per minute across order processing (50–70 ops/order), fulfillment (34–54 ops/order), lot tracking, background jobs, and concurrent user sessions (70+ warehouse + 20+ admin users).

Backup and Recovery Policy:

• Automated daily backups: 2–4 AM (off-peak window)
• Retention period: 7 days for point-in-time recovery
• Encryption: At rest (AWS KMS) and in transit (TLS 1.2+)
• Multi-AZ replication: <60 second Recovery Point Objective (RPO)
• Read replica: Enables backup testing without impacting production
• Transaction log backups: Continuous for sub-minute granularity recovery

3.6 Network and Security Architecture

All infrastructure resides within a single AWS VPC in the us-east-1 region. The primary application server accepts HTTPS traffic (port 443) from warehouse users, while the batch processing server and Apache Solr remain in private subnets accessible only within the VPC.

Key Security Controls:

• Encryption: All data encrypted at rest (AWS KMS) and in transit (TLS 1.2+)
• Database Access: RDS accessible from OFBiz servers within VPC, plus authorized external IPs via security groups
• SSH Access: Key-based authentication only via bastion host, 90-day key rotation policy
• GCP Pub/Sub Integration: Service account authentication with IAM-based access controls. Cross-cloud connectivity secured via VPN or Direct Connect for production traffic.

The network design prioritizes security without sacrificing operational efficiency — warehouse users access systems securely via HTTPS while internal components communicate over the AWS private network backbone.

3.7 Monitoring and Observability

AWS CloudWatch tracks infrastructure health and application performance. Key metrics include CPU utilization (alert at >70% sustained), memory usage (alert at >85%), database connection count (alert at >250), and replication lag (alert at >30 seconds).

Application-level monitoring tracks order import success rates (from GCP Pub/Sub), pick waving completion times, routing failures, and warehouse user response times.

3.8 Scalability and Cost Optimization

Current infrastructure operates at 40–50% capacity during peak operations and 15–20% during normal operations, providing a comfortable runway for 50–75% order volume growth without architectural changes.

Horizontal Scaling Strategy:

When vertical scaling becomes cost-ineffective or peak loads require redundancy, deploy an Elastic Load Balancer with a second primary application server. This enables zero-downtime deployments, geographic redundancy, and seasonal scaling for holiday peaks.

Cost Optimization:

• Memory-optimized database instances (r5 family) enable in-memory dataset caching, eliminating expensive disk I/O
• Compute-optimized batch server (c5 family) provides better price-performance than general-purpose instances for CPU-intensive batch jobs
• Reserved Instances (1–3 year terms) can reduce compute costs by 30–60%
• Single-region deployment (us-east-1) minimizes cross-region data transfer fees
• Provisioned IOPS for database only where needed; standard gp3 for development

4. Conclusion

The distributed deployment architecture proposed for this B2C warehouse operation demonstrates how requirements fundamentally shape infrastructure decisions. Unlike the B2B manufacturing case study where monolithic architecture efficiently served batch-processing-during-off-hours patterns, this deployment requires workload separation because batch processing runs continuously throughout the 9-hour operational window.

This case study reinforces the core methodology: structured discovery translates operational patterns into infrastructure decisions. The same discovery process produces different architectures - monolithic with off-hours batch processing for B2B e-commerce with order processing, and distributed for warehouse-heavy B2C operations with continuous operational overlap - because the underlying business models and operational patterns differ fundamentally.