How a Backend Engineer Built LinkedIn’s First Full-Scale Host Remediation System

With modern-day highly distributed and uptime-critical systems, infrastructure failure caused by software bugs or hardware aging can trigger cascading failures. For large-scale technology platforms like LinkedIn, where platform stability and developer productivity are the cornerstone of business continuity, such failures need to be addressed with speed and anticipation. Auto-remediation, or systems that self-detect and recover from failure modes, has shifted from being a best practice to a mission-critical element of engineering infrastructure.

According to reports, one of the most important players in LinkedIn's new direction toward infrastructure resilience and automation is Nikhita Kataria, a tech lead and engineering manager who created a remediation platform foundation that drives LinkedIn's capacity to rapidly address hardware and software anomalies.

Nikhita first embarked on the remediation journey towards the end of 2022 when she found repeated outages in internal tooling and production services that were resulted from host-level failures. "I still remember when some stateful teams began classifying various modes of host failures—running from faulty memory modules to naughty daemons," she recollects. Seeing a chance to automate redundant manual recovery procedures, Nikhita constructed a proof-of-concept with Airflow DAGs that automatically detected failing hosts and initiated workflows that moved applications to healthy hosts, akin to practices employed by public cloud operators to ensure SLAs. According to internal reports, the prototype subsequently grew into a full-fledged platform team that she led, ultimately providing remediation coverage for all of LinkedIn's services.

Based on the engineering timeline, Nikhita's most significant contribution was during LinkedIn's infrastructural transition to Kubernetes. As the organization started choreographing control plane services on Kubernetes, she foresaw needing to put remediation logic into this new setup starting from Day 1. Her vision made sure that even the platform in charge of coordinating other services would not be a single point of failure. "Introducing Kubernetes complexity required us to rethink remediation to be cluster-aware and policy-compliant," she explained. Nikhita spearheaded the design and implementation work, incorporating observability, health-check patterns, and safe rollback mechanisms in remediation pipelines.

Being the appointed tech lead and manager, Nikhita not only designed the core DAGs but also spearheaded a dedicated team of four engineers exclusively for remediation—a project that was her first official as an engineering manager. In addition, she focused on teaching the team object-oriented best practices, scalable config management, and clean code in order to keep the system maintainable at scale.

It made a difference. According to figures in LinkedIn's internal dashboards, the solution decreased time-to-detection of host issues to just a few minutes, and remediation latency to minutes for most cases that didn't include physical hardware replacements. This automation saved several hundred hours of manual labor every quarter and significantly lowered the risk of user-visible downtimes. The automated system could make the distinction between transient errors that could be fixed by restarting and more fundamental issues that merited full host evacuation or hardware retirement.

Yet, the journey wasn't without obstacles. Nikhita needed to steer through the complexity of supporting heterogenous workloads—stateless, stateful, batch, and real-time—each with its own architecture idiosyncrasies and failure tolerance requirements. She had to work with more than thirty partner teams, each with different timelines, which required close program management and technical coordination. "It wasn't easy to design one solution to handle so much heterogeneity, but by empathizing and demonstrating to partner teams the long-term benefit, we got buy-in," she says. According to reports, cross-functional alignment and live co-design with the other platform teams were key to getting the system into production.

When queried for input on where this area is going, Nikhita stressed the larger role AI will play in driving intelligent remediation. "With AI the buzz of the business community today, it's become incredibly simple to make inferences based on historical failure and establish an instantaneous feedback loop back into the remediation engine," she stated.

Adding to that, her combination of hands-on coding, systems design, leadership, and operational empathy provides a template for how intricate reliability engineering challenges can be approached successfully. Her counsel to young engineers in this field? "Build observability first, test with chaos, and remember that fixing a 2 a.m. alert before it wakes anyone up is the true north of infrastructure engineering."