【Linux内核】RAS（三）Intel MCA-CMCI 源码解读
【Linux内核】RAS（三）Intel MCA-CMCI 源码解读Corrected machine-check error interrupt (CMCI)是MCA的增强特性，它提供了一种threshold-based的错误上报方式。这种模式下，软件可以配置硬件corrected MC errors的阈值，硬件发生CE（Corrected Error）次数达到阈值后，会产生一个中断通知到软件处理。值得一提的是，CMCI是随MCA加入的特性，最开始只能通过软件轮询方式获取CE信息。CMCI中断通知方式的优点是每个CE都会经过IRQ Handle处理，不会丢失任一CE；而轮询方式可能因为轮询频率低、存储空间有限等原因，导致丢失CE。但是并不是说CMCI最优，CMCI的缺点是大量CE会产生中断风暴，影响机器的性能。不幸的是在云服务器场景，CE风暴是比较常见的，那么当下Intel服务器是如何解决这个问题的呢？下面会讲到。CMCI机制CMCI默认是关闭的，软件需要通过配置IA32_MCG_CAP = 1打开。软件通过IA32_MCi_CTL2 MSR来控制对应Bank使能/关闭CMCI功能。通过IA32_MCi_CTL2 Bit 14:0设置阈值，如果设置非0，则使用配置的阈值；如果CMCI不支持，则全0；CMCI机制如下图 硬件通过比较IA32_MCi_CTL2 Bit 14:0和IA32_MCi_STATUS Bit 52:38，如果数值相等，那么overflow event发送到APIC的CMCI LVT entry。如果MC error涉及多个processors，那么CMCI中断会同时发送到这些processors，比如2个cpu共享的cache发生CE，那么这两个cpu都会收到CMCI。CMCI初始化以Linux v6.3分支为例，内核使能CMCI代码
C++
arch/x86/kernel/cpu/mce/intel.c
void intel_init_cmci(void)
{
int banks;
if (!cmci_supported(&banks))
return;
mce_threshold_vector = intel_threshold_interrupt;
cmci_discover(banks);
/*
* For CPU #0 this runs with still disabled APIC, but that's
* ok because only the vector is set up. We still do another
* check for the banks later for CPU #0 just to make sure
* to not miss any events.
*/
apic_write(APIC_LVTCMCI, THRESHOLD_APIC_VECTOR|APIC_DM_FIXED);
cmci_recheck();
}
1.cmci_supported()函数主要事项包括•根据内核启动参数"mce=no_cmci,ignore_ce"判断是否打开cmci和ce上报功能•检查硬件是否支持cmci•通过MCG_CMCI_P bit判断硬件是否使能cmci功能2.mce_threshold_vector = intel_threshold_interrupt; 声明cmci的中断处理函数为intel_threshold_interrupt();3.cmci_discover()函数主要完成•遍历所有banks，通过配置IA32_MCi_CTL2寄存器使能所有bank的cmci功能；
C++
rdmsrl(MSR_IA32_MCx_CTL2(i), val);
...
val |= MCI_CTL2_CMCI_EN;
wrmsrl(MSR_IA32_MCx_CTL2(i), val);
rdmsrl(MSR_IA32_MCx_CTL2(i), val);
•设置cmci threshold值，代码如下
C++
#define CMCI_THRESHOLD 1
if (!mca_cfg.bios_cmci_threshold) {
val &= ~MCI_CTL2_CMCI_THRESHOLD_MASK;
val |= CMCI_THRESHOLD;
} else if (!(val & MCI_CTL2_CMCI_THRESHOLD_MASK)) {
/*
* If bios_cmci_threshold boot option was specified
* but the threshold is zero, we'll try to initialize
* it to 1.
*/
bios_zero_thresh = 1;
val |= CMCI_THRESHOLD;
}
如果用户未通过启动参数"mce=bios_cmci_threshold"配置值，则val = CMCI_THRESHOLD，为1；如果启动参数"mce=bios_cmci_threshold"配置，那么表示bios已配置threshold值，即val & MCI_CTL2_CMCI_THRESHOLD_MASK不为0，跳过else if判断，采用bios配置值；如果bios未配置值，val & MCI_CTL2_CMCI_THRESHOLD_MASK为0，那么驱动初始化threshold为1。4.cmci_recheck()cmci_recheck函数通过调用machine_check_poll()，检查CPU #0是否有遗漏的CE&UCE events。CMCI处理cmci中断处理函数为intel_threshold_interrupt()，定义在arch/x86/kernel/cpu/mce/intel.c
C++
/*
* The interrupt handler. This is called on every event.
* Just call the poller directly to log any events.
* This could in theory increase the threshold under high load,
* but doesn't for now.
*/
static void intel_threshold_interrupt(void)
{
if (cmci_storm_detect())
return;
machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned));
}
1.cmci_storm_detect()函数主要是对cmci storm的处理，代码如下
C++
static bool cmci_storm_detect(void)
{
unsigned int cnt = __this_cpu_read(cmci_storm_cnt);
unsigned long ts = __this_cpu_read(cmci_time_stamp);
unsigned long now = jiffies;
int r;
if (__this_cpu_read(cmci_storm_state) != CMCI_STORM_NONE)
return true;
if (time_before_eq(now, ts + CMCI_STORM_INTERVAL)) {
cnt++;
} else {
cnt = 1;
__this_cpu_write(cmci_time_stamp, now);
}
__this_cpu_write(cmci_storm_cnt, cnt);
if (cnt <= CMCI_STORM_THRESHOLD)
return false;
cmci_toggle_interrupt_mode(false);
__this_cpu_write(cmci_storm_state, CMCI_STORM_ACTIVE);
r = atomic_add_return(1, &cmci_storm_on_cpus);
mce_timer_kick(CMCI_STORM_INTERVAL);
this_cpu_write(cmci_backoff_cnt, INITIAL_CHECK_INTERVAL);
if (r == 1)
pr_notice("CMCI storm detected: switching to poll mode\n");
return true;
}
该函数通过jiffies，判断固定时间内发生的cmci次数是否大于CMCI_STORM_THRESHOLD（15），如果否则return，反之说明发生cmci storm，则执行cmci_toggle_interrupt_mode()关闭cmci功能， 切换为poll mode，通过轮询方式获取event；2.非cmci storm情况下，通过machine_check_poll(MCP_TIMESTAMP, this_cpu_ptr(&mce_banks_owned))函数获取并记录故障信息参数1定义如下，MCP_TIMESTAMP表示会记录当前TSC
C++
enum mcp_flags {
MCP_TIMESTAMP = BIT(0), /* log time stamp */
MCP_UC = BIT(1), /* log uncorrected errors */
MCP_DONTLOG = BIT(2), /* only clear, don't log */
};
machine_check_poll函数主要功能是通过读取IA32_MCG_STATUS、IA32_MCi_STATUS寄存器信息和CPU的ip、cs等相关信息，然后故障分类，将CE event或其他故障类型event记录到/dev/mcelog。用户可以通过读取/dev/mcelog获取错误记录。执行流程如下，过程说明在代码注释中
C++
bool machine_check_poll(enum mcp_flags flags, mce_banks_t *b)
{
if (flags & MCP_TIMESTAMP)
m.tsc = rdtsc(); // 记录当前TSC
/*CE Error记录*/
/* If this entry is not valid, ignore it */
if (!(m.status & MCI_STATUS_VAL))
continue;
/*
* If we are logging everything (at CPU online) or this
* is a corrected error, then we must log it.
*/
if ((flags & MCP_UC) || !(m.status & MCI_STATUS_UC))
goto log_it;
/*UCNA Error记录*/
/*
* Log UCNA (SDM: 15.6.3 "UCR Error Classification")
* UC == 1 && PCC == 0 && S == 0
*/
if (!(m.status & MCI_STATUS_PCC) && !(m.status & MCI_STATUS_S))
goto log_it;
/*通过mce_log记录故障信息*/
log_it:
/*
* Don't get the IP here because it's unlikely to
* have anything to do with the actual error location.
*/
if (!(flags & MCP_DONTLOG) && !mca_cfg.dont_log_ce)
mce_log(&m);
else if (mce_usable_address(&m)) {
/*
* Although we skipped logging this, we still want
* to take action. Add to the pool so the registered
* notifiers will see it.
*/
if (!mce_gen_pool_add(&m))
mce_schedule_work();
}
}
总结一下，CMCI是MCA的一个增强特性，主要用于将硬件CE、UCNA等类型故障通过中断方式上报到软件，软件收到中断后，执行中断处理函数intel_threshold_interrupt()采取irq mode或poll mode记录错误信息到/dev/mcelog，用户态可以通过/dev/mcelog获取硬件故障信息。参考文档：《Intel® 64 and IA-32 Architectures Software Developer’s Manual 》
页:
[1]