在自有模块的处理中，我们设计了一个内核线程去做gc，
但同时，我们又用到了rcu，rcu中也会去抢gc的锁，由于该锁用的spin_lock，而不是spin_lock_bh,并没有关软中断，所以在rcu上下文中拿不到锁，造成死锁。
[106251.128106] NMI watchdog: BUG: soft lockup - CPU# stuck for 23s! [kflow_gcd:]
[106251.129425] Modules linked in: newsendfile(OE) witdriver(OE) mysendmsg(OE) xfs libcrc32c fuse tipc(OE) ossmod(OE) iptable_filter mptctl mptbase bonding dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure joydev sg mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_cpufreq acpi_power_meter tcp_bbr sch_fq binfmt_misc ip_tables ext4 mbcache jbd2 raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ahci libahci libata
[106251.129481] ixgbe(OE) i40e(OE) mpt3sas(OE) ptp raid_class pps_core scsi_transport_sas i2c_core dca [last unloaded: witdriver]
[106251.129492] CPU: PID: Comm: kflow_gcd Tainted: G W OEL ------------ 3.10.-693.21..el7.x86_64 #
[106251.129493] Hardware name: ZTE ZXCDN/SC621DI-16F, BIOS .0b //
[106251.129496] task: ffff8839a2e70000 ti: ffff884f3fb34000 task.ti: ffff884f3fb34000
[106251.129498] RIP: :[<ffffffff811005ce>] [<ffffffff811005ce>] native_queued_spin_lock_slowpath+0x1ce/0x200
[106251.129508] RSP: :ffff885fbdf43dd0 EFLAGS:
[106251.129510] RAX: RBX: 000000000000060c RCX:
[106251.129512] RDX: RSI: RDI: ffff885fbdf59b60
[106251.129513] RBP: ffff885fbdf43dd0 R08: R09:
[106251.129515] R10: ffff88607fbc84a0 R11: ffffea00fba2ae00 R12: ffff885fbdf43d48
[106251.129517] R13: ffffffff816c6732 R14: ffff885fbdf43dd0 R15: ffff8852e87ccf40
[106251.129519] FS: () GS:ffff885fbdf40000() knlGS:
[106251.129521] CS: DS: ES: CR0:
[106251.129523] CR2: 00007f582cb9816c CR3: 0000000001a0a000 CR4: 00000000003607e0
[106251.129525] DR0: DR1: DR2:
[106251.129527] DR3: DR6: 00000000fffe0ff0 DR7:
[106251.129528] Call Trace:
[106251.129530] <IRQ> [106251.129539] [<ffffffff816adeee>] queued_spin_lock_slowpath+0xb/0xf
[106251.129546] [<ffffffff816bb080>] _raw_spin_lock+0x20/0x30
[106251.129555] [<ffffffffc055a3e5>] wit_inode_replace+0x3b5/0x400 [witdriver]
[106251.129561] [<ffffffff8119243f>] ? free_pages.part.+0x3f/0x50
[106251.129567] [<ffffffffc055a534>] free_wit_flow+0x104/0x150 [witdriver]----------------rcu中调用
[106251.129571] [<ffffffffc055a592>] release_wit_flow+0x12/0x20 [witdriver]
[106251.129577] [<ffffffff81140340>] rcu_process_callbacks+0x1e0/0x580
[106251.129583] [<ffffffff8109404d>] __do_softirq+0xfd/0x290
[106251.129588] [<ffffffff816c8afc>] call_softirq+0x1c/0x30
[106251.129594] [<ffffffff8102d435>] do_softirq+0x65/0xa0
[106251.129597] [<ffffffff81094495>] irq_exit+0x175/0x180
[106251.129600] [<ffffffff816c9e88>] smp_apic_timer_interrupt+0x48/0x60
[106251.129603] [<ffffffff816c6732>] apic_timer_interrupt+0x162/0x170
[106251.129604] <EOI> [106251.129613] [<ffffffff81347655>] ? __list_del_entry+0x35/0xd0
[106251.129615] [<ffffffff813476fd>] list_del+0xd/0x30
[106251.129620] [<ffffffffc055fdea>] prune_wait_free_list+0x9a/0x1c0 [witdriver]----------sys中调用
[106251.129624] [<ffffffffc055ff65>] kflow_gcd_fn+0x55/0x1e0 [witdriver]
[106251.129628] [<ffffffffc055ff10>] ? prune_wait_free_list+0x1c0/0x1c0 [witdriver]
[106251.129634] [<ffffffff810b5241>] kthread+0xd1/0xe0
[106251.129637] [<ffffffff810b5170>] ? insert_kthread_work+0x40/0x40
[106251.129641] [<ffffffff816c5577>] ret_from_fork+0x77/0xb0
[106251.129644] [<ffffffff810b5170>] ? insert_kthread_work+0x40/0x40
void wit_unref_flow(void *flow)
{
struct wit_fq_flow *f = (struct wit_fq_flow*)flow;
if ( !witdriver_init_done || !f ) return;
if ( atomic_dec_and_test(&(f->users)) )
{
call_rcu(&f->rcu_head, release_wit_flow);
}
}
由于call_rcu其实是在软中断中完成的，
crash> irq -b
SOFTIRQ_VEC ACTION
[] ffffffff81092e40 <tasklet_hi_action>
[] ffffffff8109bcf0 <run_timer_softirq>
[] ffffffff81590d60 <net_tx_action>
[] ffffffff81592b80 <net_rx_action>
[] ffffffff81308250 <blk_done_softirq>
[] ffffffff81358f00 <irq_poll_softirq>
[] ffffffff81092d00 <tasklet_action>
[] ffffffff810d6610 <run_rebalance_domains>
[] ffffffff8113ea70 <rcu_process_callbacks>
release_wit_flow 会和gc流程抢一把锁，当gc流程拿了锁又被中断之后，执行到rcu的流程，执行 release_wit_flow 去抢锁，造成死锁。
所以需要加bh。
需要注意的是，这个软中断打印，很明显缺少[8],这个8对应的是啥呢？
crash> p softirq_to_name
softirq_to_name = $ =
{0xffffffff8195816a "HI", 0xffffffff818f79c0 "TIMER", 0xffffffff81914a4c "NET_TX", 0xffffffff81914a53 "NET_RX", 0xffffffff819547d4 "BLOCK", 0xffffffff81914a8a "BLOCK_IOPOLL", 0xffffffff81914a63 "TASKLET", 0xffffffff81914a6b "SCHED", 0xffffffff81914a71 "HRTIMER", 0xffffffff81914a79 "RCU"}
看起来【8】对应的是 HRTIMER ，而 HRTIMER 其实是在硬中断中实现的，而不是在软中断中实现的，所以这个看起来应该是废弃了。
hrtimers - High-resolution kernel timers，在rh的3.10内核中，高精度定时器是通过硬中断来实现的，
void __init hrtimers_init(void)
{
hrtimer_cpu_notify(&hrtimers_nb, (unsigned long)CPU_UP_PREPARE,
(void *)(long)smp_processor_id());
register_cpu_notifier(&hrtimers_nb);
}
对应的中断处理堆栈可以参考如下：
[ 6401.459859] [<ffffffff816b9d53>] _raw_spin_lock_bh+0x33/0x40
[ 6401.466087] [<ffffffffc0358971>] wit_hrtimer_pool_notify+0x181/0x3c0 [witdriver]-----这个是我们的timer的func
[ 6401.474060] [<ffffffffc03587f0>] ? wit_send_tasklet+0x590/0x590 [witdriver]
[ 6401.481600] [<ffffffff810b9096>] __hrtimer_run_queues+0xd6/0x260
[ 6401.488187] [<ffffffff810b962f>] hrtimer_interrupt+0xaf/0x1d0
[ 6401.494566] [<ffffffff8105467b>] local_apic_timer_interrupt+0x3b/0x60
[ 6401.501625] [<ffffffff816c8e83>] smp_apic_timer_interrupt+0x43/0x60
[ 6401.508511] [<ffffffff816c5732>] apic_timer_interrupt+0x162/0x170