Adaptive Replica Placement for Hot Embedding Vectors in Distributed Recommendation Serving Systems for E-Commerce Marketplaces

• Rifki Ardana Putra

  Institut Teknologi Nusantara Madani, Department of Informatics Engineering, Jalan Lingkar Selatan, Tamantirto, Bantul, Yogyakarta, Indonesia

  Author

• Galang Prasetya Nugraha

  Universitas Borneo Samudra, Faculty of Computer Science and Information Technology, Jalan Kuaro, Samarinda Ulu, East Kalimantan, Indonesia

  Author

Large-scale e-commerce marketplaces rely on distributed recommendation serving systems that retrieve and aggregate embedding vectors under strict tail-latency and availability targets. In these systems, a small fraction of embedding vectors often accounts for a disproportionate share of lookup traffic, and the identity of this hot set can shift rapidly with promotions, inventory changes, and seasonal demand. Replica placement for embedding vectors is therefore a central operational lever, yet it is constrained by memory budgets, network topology, update propagation costs, and multi-tenant isolation requirements. This paper studies adaptive replica placement for hot embedding vectors in distributed recommendation serving, focusing on the gap between static sharding approaches and the non-stationary, bursty workloads typical of marketplaces. We formulate replica placement as an online optimization problem that jointly minimizes expected tail latency and inter-rack traffic while enforcing per-node memory limits and bounded staleness for mutable embeddings. The proposed approach combines streaming hotness estimation, topology-aware placement, and a stability mechanism that prevents oscillations during demand surges. We develop a mathematical model linking request routing, replica multiplicity, and queueing delay, and we derive an adaptive control policy that updates placements using constrained gradients computed from live telemetry. The design addresses practical concerns including write amplification, cache warmup, partial failures, and fairness across embedding tables. The result is an implementable framework that improves latency robustness under shifting hot sets without relying on static overprovisioning or manual tuning.

• PDF