Intel Xeon Server CPU Maximum Wake Latency Measurement
Govtva, Vladislav (2019)
2019
All rights reserved. This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.
Julkaisun pysyvä osoite on
https://urn.fi/URN:NBN:fi:amk-201905139461
https://urn.fi/URN:NBN:fi:amk-201905139461
Tiivistelmä
Modern CPUs implement mechanisms for managing power consumption when being idle by means of clock- or power-gating various internal blocks. These idle states are called "C-states". If a CPU is in an idle C-state to save power but there is a task ready to be executed, the CPU exits the C-state, although with delay. This delay is referred to as "C-state wake latency".
There are typically multiple C-state levels. And each level has its own wake latency. Based on those latencies Linux will choose which level to use in time-sensitive applications. The aim of this research is to measure this latency with different levels of idle states. The paper will explore idle power states with different generations of Intel Server CPUs, how those states affect wake latency, as well as how Linux performs power management. There are requirements for the C-state wake latency measurement. The test must measure with greater than 95% precision and with sufficient resolution to measure 1-2 µs wake latency. The measurement method should not require expensive or unobtainable equipment, such as custom-made motherboards with measurement points.
The test developed for measuring C-state wake latency uses a special Network Interface Controller (NIC). NICs’ architecture and high resolution clock allow it to send a network packet at scheduled time with 32 nanosecond granularity. The NIC keeps track of those network packets, which allows the CPU to stay idle until a packet is sent. When the packet is sent, NIC notifies the CPU by issuing an interrupt. This interrupt causes the CPU to wake up and take a timestamp with special software. The difference between the interrupt time and the timestamp is C-state wake latency.
The results showed that there is significant difference between generations of Intel Server CPUs and Linux’s expectations for certain C-state’s maximum wake latencies were incorrect.
There are typically multiple C-state levels. And each level has its own wake latency. Based on those latencies Linux will choose which level to use in time-sensitive applications. The aim of this research is to measure this latency with different levels of idle states. The paper will explore idle power states with different generations of Intel Server CPUs, how those states affect wake latency, as well as how Linux performs power management. There are requirements for the C-state wake latency measurement. The test must measure with greater than 95% precision and with sufficient resolution to measure 1-2 µs wake latency. The measurement method should not require expensive or unobtainable equipment, such as custom-made motherboards with measurement points.
The test developed for measuring C-state wake latency uses a special Network Interface Controller (NIC). NICs’ architecture and high resolution clock allow it to send a network packet at scheduled time with 32 nanosecond granularity. The NIC keeps track of those network packets, which allows the CPU to stay idle until a packet is sent. When the packet is sent, NIC notifies the CPU by issuing an interrupt. This interrupt causes the CPU to wake up and take a timestamp with special software. The difference between the interrupt time and the timestamp is C-state wake latency.
The results showed that there is significant difference between generations of Intel Server CPUs and Linux’s expectations for certain C-state’s maximum wake latencies were incorrect.