Modern On-Line Data Intensive (OLDI) applications have evolved from monolithic systems to instead comprise numerous, distributed microservices interacting via Remote Procedure Calls (RPCs). Microservices face sub-ms RPC latency goals, much tighter than their monolithic ancestors that must meet >=100ms latency targets. Sub-ms-scale threading and on currency design effects as well as OS and network overheads that were once insignificant for such monoliths, can now come to dominate in the sub-ms-scale microservice regime. It is therefore vital to characterize the influence of threading design, OS, and network effects on microservices. Unfortunately, widely used academic data center benchmark suites are unsuitable to aid this characterization as they use monolithic rather than microservice architectures.

We first investigate how OS/network overheads impact microservice tail latency by developing a complete suite of microservices called mSuite that we use to facilitate our study. Our characterization reveals that the relationship between optimal OS/network parameters and service load is complex. Our primary finding is that non-optimal OS scheduler decisions can degrade microservice tail latency by up to ~87%.

Secondly, we investigate how threading design critically impacts microservice tail latency by developing a taxonomy of threading models – a structured understanding of the implications of how microservices manage concurrency and interact with RPC interfaces under wide-ranging loads. We develop mTune, a system that has two features: (1) a novel framework that abstracts threading model implementation from application code, and (2) a novel automatic load adaptation system that curtails microservice tail latency by exploiting inherent latency trade-offs revealed in our taxonomy to transition among threading models. We study mTune in the context of mSuite to demonstrate up to 1.9x tail latency improvements over static threading choices and state-of-the-art adaptation techniques.