The Memory Wall in Multi-Categorical Feature Models

Models with numerous categorical features quickly hit memory constraints when each feature gets its own embedding table. Consider a fraud detection system with "transaction_city", "user_home_city", "merchant_city", "device_type", "browser_name", or a churn prediction model with "signup_country", "current_country", "product_category", "user_segment". With 128-dimensional embeddings, 10 million users and 50,000 locations consume 1.3GB for users alone, plus 6.4GB for three city-type features if treated independently. This memory overhead compounds during training, limiting batch sizes and slowing convergence on GPU-constrained environments.

Semantic Grouping: When to Share Embeddings

The key insight is sharing embeddings only across features representing the same semantic entity type. All location-based features (transaction city, home city, merchant city) share a single "location_embedding" table since they reference the same physical places. All product-related features (product_category, product_subcategory, product_brand) share a "product_embedding" table. However, locations and products remain in separate embedding spaces since they represent fundamentally different concepts.

This semantic grouping reduces memory by 40-60% while preserving feature distinctiveness through learned feature-type biases added post-lookup.

Performance Impact: Convergence and Generalization

Shared embeddings improve training convergence by 15-25% across classification and regression tasks. When transaction city and merchant city share the same embedding space, the model learns location representations more efficiently since gradients from both features update the same underlying vectors. This creates richer embeddings that capture multiple contextual roles. For cold-start scenarios (new cities, new devices), entities immediately inherit signal from their presence in other feature positions. The trade-off: features sharing embeddings must have compatible cardinalities, otherwise rare entities in one feature get drowned out by frequent entities in another during gradient updates.

Applied Insight: Monitoring Embedding Health

Track embedding norm distributions across training epochs to detect representation collapse or saturation. Healthy embeddings maintain norms between 0.5 and 2.0 with gradual stabilization. Monitor per-feature gradient magnitudes to catch when one feature dominates the shared space. Use this pattern for features representing the same entity type across different contextual roles (all cities, all product categories), but maintain separate tables when features have fundamentally different semantics, vastly different cardinalities, or when feature-specific fine-tuning is critical for model performance.