南寨小子 Help

TCP重传导致连接Reset

TCP重传8次还未收到ACK,会发送RST中断连接,如下结合技术原理及案例进行分析。

1. timer初始化

在创建socket的过程中会初始化TCP重传定时器,详细过程见下图。

Tcp retransmission timer
    void tcp_init_xmit_timers(struct sock *sk) { inet_csk_init_xmit_timers(sk, &tcp_write_timer, &tcp_delack_timer, &tcp_keepalive_timer); hrtimer_init(&tcp_sk(sk)->pacing_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED_SOFT); tcp_sk(sk)->pacing_timer.function = tcp_pace_kick; hrtimer_init(&tcp_sk(sk)->compressed_ack_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL_PINNED_SOFT); tcp_sk(sk)->compressed_ack_timer.function = tcp_compressed_ack_kick; }
    /* * Using different timers for retransmit, delayed acks and probes * We may wish use just one timer maintaining a list of expire jiffies * to optimize. */ void inet_csk_init_xmit_timers(struct sock *sk, void (*retransmit_handler)(struct timer_list *t), void (*delack_handler)(struct timer_list *t), void (*keepalive_handler)(struct timer_list *t)) { struct inet_connection_sock *icsk = inet_csk(sk); timer_setup(&icsk->icsk_retransmit_timer, retransmit_handler, 0); timer_setup(&icsk->icsk_delack_timer, delack_handler, 0); timer_setup(&sk->sk_timer, keepalive_handler, 0); icsk->icsk_pending = icsk->icsk_ack.pending = 0; }

    2. tcp_retries2

    内核文档中对tcp_retries2的描述如下,其中提到RFC 1122建议超时时间最少为100秒,结合TCP_RTO_MINTCP_RTO_MAX以及exponential backoff机制,得出相应的重传次数为8

    TCP重传机制采用指数退避(exponential backoff )算法来计算重传间隔。初始RTO(Retransmission Timeout)为TCP_RTO_MIN(200ms),每次重传失败后RTO翻倍,直到达到TCP_RTO_MAX(120秒)。通过这种方式,前8次重传的累计时间约为: 200ms + 400ms + 800ms + 1.6s + 3.2s + 6.4s + 12.8s + 25.6s ≈ 51秒 ,第9次重传要间隔51.2s ,如果重传9次总时间将超过100秒,不满足RFC 1122规定的100秒最低要求,因此在Linux实现中重传8次后发送RST中断连接。

    #define TCP_RTO_MAX ((unsigned)(120*HZ)) #define TCP_RTO_MIN ((unsigned)(HZ/5))
    // /home/kangxiaoning/workspace/kernel-4.19.90-2404.2.0/Documentation/networking/ip-sysctl.txt tcp_retries2 - INTEGER This value influences the timeout of an alive TCP connection, when RTO retransmissions remain unacknowledged. Given a value of N, a hypothetical TCP connection following exponential backoff with an initial RTO of TCP_RTO_MIN would retransmit N times before killing the connection at the (N+1)th RTO. The default value of 15 yields a hypothetical timeout of 924.6 seconds and is a lower bound for the effective timeout. TCP will effectively time out at the first RTO which exceeds the hypothetical timeout. RFC 1122 recommends at least 100 seconds for the timeout, which corresponds to a value of at least 8.

    3. 重传定时器执行过程

    根据第1步了解到每个TCP socket在创建阶段都设置了定时器,沿着定时器调用逻辑分析处理过程。

    TCP使用定时器机制来触发重传。当数据包发送后没有收到确认时,重传定时器会触发tcp_write_timer函数。该函数会检查socket的锁状态,如果socket未被用户占用,则直接调用tcp_write_timer_handler处理;否则通过设置TCP_WRITE_TIMER_DEFERRED标志位延迟处理,避免锁竞争。

    static void tcp_write_timer(struct timer_list *t) { struct inet_connection_sock *icsk = from_timer(icsk, t, icsk_retransmit_timer); struct sock *sk = &icsk->icsk_inet.sk; bh_lock_sock(sk); if (!sock_owned_by_user(sk)) { tcp_write_timer_handler(sk); } else { /* delegate our work to tcp_release_cb() */ if (!test_and_set_bit(TCP_WRITE_TIMER_DEFERRED, &sk->sk_tsq_flags)) sock_hold(sk); } bh_unlock_sock(sk); sock_put(sk); }

    如下是TCP不同定时器的处理函数,重传对应的是tcp_retransmit_timer(sk)

    tcp_write_timer_handler根据icsk_pending字段区分不同类型的定时器事件。 ICSK_TIME_RETRANS表示重传定时器到期,此时会调用tcp_retransmit_timer执行实际的重传逻辑。其他定时器类型包括:REO_TIMEOUT(重排序超时)、LOSS_PROBE(丢包探测)和PROBE0(零窗口探测)。

    void tcp_write_timer_handler(struct sock *sk) { struct inet_connection_sock *icsk = inet_csk(sk); int event; if (((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN)) || !icsk->icsk_pending) goto out; if (time_after(icsk->icsk_timeout, jiffies)) { sk_reset_timer(sk, &icsk->icsk_retransmit_timer, icsk->icsk_timeout); goto out; } tcp_mstamp_refresh(tcp_sk(sk)); event = icsk->icsk_pending; switch (event) { case ICSK_TIME_REO_TIMEOUT: tcp_rack_reo_timeout(sk); break; case ICSK_TIME_LOSS_PROBE: tcp_send_loss_probe(sk); break; case ICSK_TIME_RETRANS: icsk->icsk_pending = 0; tcp_retransmit_timer(sk); break; case ICSK_TIME_PROBE0: icsk->icsk_pending = 0; tcp_probe_timer(sk); break; } out: sk_mem_reclaim(sk); }

    tcp_retransmit_timer是重传机制的核心函数。它首先处理特殊情况(如Fast Open、ZeroWindow),然后调用tcp_write_timeout检查是否达到重传次数上限。如果未超时,则进入丢包状态(tcp_enter_loss),尝试重传队列头部的数据包,并根据指数退避算法更新下次重传时间(icsk_rto翻倍,最大不超过TCP_RTO_MAX)。icsk_backoff和icsk_retransmits分别记录退避次数和重传次数。

    void tcp_retransmit_timer(struct sock *sk) { struct tcp_sock *tp = tcp_sk(sk); struct net *net = sock_net(sk); struct inet_connection_sock *icsk = inet_csk(sk); if (tp->fastopen_rsk) { WARN_ON_ONCE(sk->sk_state != TCP_SYN_RECV && sk->sk_state != TCP_FIN_WAIT1); tcp_fastopen_synack_timer(sk); /* Before we receive ACK to our SYN-ACK don't retransmit * anything else (e.g., data or FIN segments). */ return; } if (!tp->packets_out || WARN_ON_ONCE(tcp_rtx_queue_empty(sk))) return; tp->tlp_high_seq = 0; if (!tp->snd_wnd && !sock_flag(sk, SOCK_DEAD) && !((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV))) { /* Receiver dastardly shrinks window. Our retransmits * become zero probes, but we should not timeout this * connection. If the socket is an orphan, time it out, * we cannot allow such beasts to hang infinitely. */ struct inet_sock *inet = inet_sk(sk); if (sk->sk_family == AF_INET) { net_dbg_ratelimited("Peer %pI4:%u/%u unexpectedly shrunk window %u:%u (repaired)\n", &inet->inet_daddr, ntohs(inet->inet_dport), inet->inet_num, tp->snd_una, tp->snd_nxt); } #if IS_ENABLED(CONFIG_IPV6) else if (sk->sk_family == AF_INET6) { net_dbg_ratelimited("Peer %pI6:%u/%u unexpectedly shrunk window %u:%u (repaired)\n", &sk->sk_v6_daddr, ntohs(inet->inet_dport), inet->inet_num, tp->snd_una, tp->snd_nxt); } #endif if (tcp_jiffies32 - tp->rcv_tstamp > TCP_RTO_MAX) { tcp_write_err(sk); goto out; } tcp_enter_loss(sk); tcp_retransmit_skb(sk, tcp_rtx_queue_head(sk), 1); __sk_dst_reset(sk); goto out_reset_timer; } __NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPTIMEOUTS); if (tcp_write_timeout(sk)) goto out; if (icsk->icsk_retransmits == 0) { int mib_idx = 0; if (icsk->icsk_ca_state == TCP_CA_Recovery) { if (tcp_is_sack(tp)) mib_idx = LINUX_MIB_TCPSACKRECOVERYFAIL; else mib_idx = LINUX_MIB_TCPRENORECOVERYFAIL; } else if (icsk->icsk_ca_state == TCP_CA_Loss) { mib_idx = LINUX_MIB_TCPLOSSFAILURES; } else if ((icsk->icsk_ca_state == TCP_CA_Disorder) || tp->sacked_out) { if (tcp_is_sack(tp)) mib_idx = LINUX_MIB_TCPSACKFAILURES; else mib_idx = LINUX_MIB_TCPRENOFAILURES; } if (mib_idx) __NET_INC_STATS(sock_net(sk), mib_idx); } tcp_enter_loss(sk); if (tcp_retransmit_skb(sk, tcp_rtx_queue_head(sk), 1) > 0) { /* Retransmission failed because of local congestion, * do not backoff. */ if (!icsk->icsk_retransmits) icsk->icsk_retransmits = 1; inet_csk_reset_xmit_timer(sk, ICSK_TIME_RETRANS, min(icsk->icsk_rto, TCP_RESOURCE_PROBE_INTERVAL), TCP_RTO_MAX); goto out; } /* Increase the timeout each time we retransmit. Note that * we do not increase the rtt estimate. rto is initialized * from rtt, but increases here. Jacobson (SIGCOMM 88) suggests * that doubling rto each time is the least we can get away with. * In KA9Q, Karn uses this for the first few times, and then * goes to quadratic. netBSD doubles, but only goes up to *64, * and clamps at 1 to 64 sec afterwards. Note that 120 sec is * defined in the protocol as the maximum possible RTT. I guess * we'll have to use something other than TCP to talk to the * University of Mars. * * PAWS allows us longer timeouts and large windows, so once * implemented ftp to mars will work nicely. We will have to fix * the 120 second clamps though! */ icsk->icsk_backoff++; icsk->icsk_retransmits++; out_reset_timer: /* If stream is thin, use linear timeouts. Since 'icsk_backoff' is * used to reset timer, set to 0. Recalculate 'icsk_rto' as this * might be increased if the stream oscillates between thin and thick, * thus the old value might already be too high compared to the value * set by 'tcp_set_rto' in tcp_input.c which resets the rto without * backoff. Limit to TCP_THIN_LINEAR_RETRIES before initiating * exponential backoff behaviour to avoid continue hammering * linear-timeout retransmissions into a black hole */ if (sk->sk_state == TCP_ESTABLISHED && (tp->thin_lto || READ_ONCE(net->ipv4.sysctl_tcp_thin_linear_timeouts)) && tcp_stream_is_thin(tp) && icsk->icsk_retransmits <= TCP_THIN_LINEAR_RETRIES) { icsk->icsk_backoff = 0; icsk->icsk_rto = clamp(__tcp_set_rto(tp), tcp_rto_min(sk), TCP_RTO_MAX); } else { /* Use normal (exponential) backoff */ icsk->icsk_rto = min(icsk->icsk_rto << 1, TCP_RTO_MAX); } inet_csk_reset_xmit_timer(sk, ICSK_TIME_RETRANS, tcp_clamp_rto_to_user_timeout(sk), TCP_RTO_MAX); if (retransmits_timed_out(sk, READ_ONCE(net->ipv4.sysctl_tcp_retries1) + 1, 0)) __sk_dst_reset(sk); out:; }

    tcp_retransmit_timer(sk)中调用到tcp_write_timeout()tcp_write_timeout()实现了重传8次和发送RST包的逻辑。

    tcp_write_timeout是判断连接是否应该被终止的关键函数。对于orphaned socket(SOCK_DEAD标志位被设置,表示应用层已经关闭),会调用tcp_orphan_retries获取最大重传次数。如果重传次数达到上限,会调用tcp_out_of_resources检查是否需要发送RST包。这个机制防止orphaned socket无限占用系统资源。

    static int tcp_write_timeout(struct sock *sk) { struct inet_connection_sock *icsk = inet_csk(sk); struct tcp_sock *tp = tcp_sk(sk); struct net *net = sock_net(sk); bool expired = false, do_reset; int retry_until; if ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) { if (icsk->icsk_retransmits) { dst_negative_advice(sk); } else { sk_rethink_txhash(sk); } retry_until = icsk->icsk_syn_retries ? : net->ipv4.sysctl_tcp_syn_retries; expired = icsk->icsk_retransmits >= retry_until; } else { if (retransmits_timed_out(sk, READ_ONCE(net->ipv4.sysctl_tcp_retries1), 0)) { /* Black hole detection */ tcp_mtu_probing(icsk, sk); dst_negative_advice(sk); } else { sk_rethink_txhash(sk); } retry_until = READ_ONCE(net->ipv4.sysctl_tcp_retries2); if (sock_flag(sk, SOCK_DEAD)) { const bool alive = icsk->icsk_rto < TCP_RTO_MAX; // 获取到retry_until为8 retry_until = tcp_orphan_retries(sk, alive); do_reset = alive || !retransmits_timed_out(sk, retry_until, 0); // 在这里发送了RST包 if (tcp_out_of_resources(sk, do_reset)) return 1; } } if (!expired) expired = retransmits_timed_out(sk, retry_until, icsk->icsk_user_timeout); tcp_fastopen_active_detect_blackhole(sk, expired); if (BPF_SOCK_OPS_TEST_FLAG(tp, BPF_SOCK_OPS_RTO_CB_FLAG)) tcp_call_bpf_3arg(sk, BPF_SOCK_OPS_RTO_CB, icsk->icsk_retransmits, icsk->icsk_rto, (int)expired); if (expired) { /* Has it gone just too far? */ tcp_write_err(sk); return 1; } return 0; }

    retry_until = tcp_orphan_retries(sk, alive);中获取retry_until8 ,即重传8次

    tcp_orphan_retries函数的设计体现了TCP的防御性编程思想。对于"活跃"的orphaned socket(alive参数为true,表示RTO还未达到最大值),即使sysctl_tcp_orphan_retries为0,也会返回8作为重传次数。这个魔数8的选择基于这样的计算:以最小RTO 200ms开始,经过8次指数退避重传,总时间超过100秒,满足RFC 1122的最低要求,同时避免过长时间占用资源。

    /** * tcp_orphan_retries() - Returns maximal number of retries on an orphaned socket * @sk: Pointer to the current socket. * @alive: bool, socket alive state */ static int tcp_orphan_retries(struct sock *sk, bool alive) { int retries = READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_orphan_retries); /* May be zero. */ /* We know from an ICMP that something is wrong. */ if (sk->sk_err_soft && !alive) retries = 0; /* However, if socket sent something recently, select some safe * number of retries. 8 corresponds to >100 seconds with minimal * RTO of 200msec. */ if (retries == 0 && alive) retries = 8; return retries; }

    4. Reset发送逻辑

    tcp_out_of_resources函数实现了TCP的资源保护机制。当orphaned socket达到重传上限时,该函数会评估是否需要发送RST包来强制关闭连接。发送RST的条件包括:最近发送过数据(防止长时间静默的连接)或接收窗口为0(对端可能已经崩溃)。这种机制虽然违反了TCP规范的某些要求,但对于防止DoS攻击和资源耗尽是必要的。

    /** * tcp_out_of_resources() - Close socket if out of resources * @sk: pointer to current socket * @do_reset: send a last packet with reset flag * * Do not allow orphaned sockets to eat all our resources. * This is direct violation of TCP specs, but it is required * to prevent DoS attacks. It is called when a retransmission timeout * or zero probe timeout occurs on orphaned socket. * * Also close if our net namespace is exiting; in that case there is no * hope of ever communicating again since all netns interfaces are already * down (or about to be down), and we need to release our dst references, * which have been moved to the netns loopback interface, so the namespace * can finish exiting. This condition is only possible if we are a kernel * socket, as those do not hold references to the namespace. * * Criteria is still not confirmed experimentally and may change. * We kill the socket, if: * 1. If number of orphaned sockets exceeds an administratively configured * limit. * 2. If we have strong memory pressure. * 3. If our net namespace is exiting. */ static int tcp_out_of_resources(struct sock *sk, bool do_reset) { struct tcp_sock *tp = tcp_sk(sk); int shift = 0; /* If peer does not open window for long time, or did not transmit * anything for long time, penalize it. */ if ((s32)(tcp_jiffies32 - tp->lsndtime) > 2*TCP_RTO_MAX || !do_reset) shift++; /* If some dubious ICMP arrived, penalize even more. */ if (sk->sk_err_soft) shift++; if (tcp_check_oom(sk, shift)) { /* Catch exceptional cases, when connection requires reset. * 1. Last segment was sent recently. */ if ((s32)(tcp_jiffies32 - tp->lsndtime) <= TCP_TIMEWAIT_LEN || /* 2. Window is closed. */ (!tp->snd_wnd && !tp->packets_out)) do_reset = true; if (do_reset) tcp_send_active_reset(sk, GFP_ATOMIC); tcp_done(sk); __NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONMEMORY); return 1; } if (!check_net(sock_net(sk))) { /* Not possible to send reset; just close */ tcp_done(sk); return 1; } return 0; }

    tcp_send_active_reset构造并发送RST包。该函数分配一个最小的skb(只包含TCP头部),设置RST和ACK标志位,使用当前可接受的序列号 ,然后通过tcp_transmit_skb发送。这个RST包不会被重传,是单向通知对端连接已经被强制关闭。这遵循了RFC 2525的建议,用于处理异常情况下的连接清理。

    /* We get here when a process closes a file descriptor (either due to * an explicit close() or as a byproduct of exit()'ing) and there * was unread data in the receive queue. This behavior is recommended * by RFC 2525, section 2.17. -DaveM */ void tcp_send_active_reset(struct sock *sk, gfp_t priority) { struct sk_buff *skb; TCP_INC_STATS(sock_net(sk), TCP_MIB_OUTRSTS); /* NOTE: No TCP options attached and we never retransmit this. */ skb = alloc_skb(MAX_TCP_HEADER, priority); if (!skb) { NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTFAILED); return; } /* Reserve space for headers and prepare control bits. */ skb_reserve(skb, MAX_TCP_HEADER); tcp_init_nondata_skb(skb, tcp_acceptable_seq(sk), TCPHDR_ACK | TCPHDR_RST); tcp_mstamp_refresh(tcp_sk(sk)); /* Send it off. */ if (tcp_transmit_skb(sk, skb, 0, priority)) NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTFAILED); /* skb of trace_tcp_send_reset() keeps the skb that caused RST, * skb here is different to the troublesome skb, so use NULL */ trace_tcp_send_reset(sk, NULL); }

    5. 案例

    5.1 问题现象

    1. ECS访问NAS, message日志中出现nfs: server xxx not responding, timed out报错。

    2. 监控显示ECS的TCP每分钟重传高达200-1000

    5.2 问题分析

    通过重现问题及抓包,根据时间先后整理,异常过程大概如下:

    1. NAS端频繁发送zero window

    2. ECS在某个包重传了8次后发送了RST中断连接,随后向NAS发起了TCP三次握手

    3. 在message日志中出现not responding报错,用户进入NAS目录执行ls命令卡住

    4. ECS尝试发送SYN包建立连接,重传6次没有收到NAS端SYN+ACK包,该过程持续了3次均失败

    5. 第4次连接建立成功,NAS访问恢复

    5.3 解决方案

    1. 降低应用并发减少I/O负载可解决问题,通过时间换取可靠性

    2. 使用nconnect挂载选项,通过多个连接提升NAS处理能力,要求linux kernel versions >= 5.3

    mount -t nfs -o ro,nconnect=16 198.18.0.100:/datasets /mnt/datasets
    1. 缓解方案,修改内核参数sunrpc.tcp_max_slot_table_entries为256。

      • 验证 :TCP重传仍然较多,但是出现nsf: server xxx not respondint, timed out报错机率降低。

      • 分析 :调整的参数并没有降低生产者生产能力,只是在生产者和消费者中间链路的OS上限制了并发,在OS缓冲了一下,把瓶颈点转移到OS了。应用负载没变,NAS处理能力没变,整个链路瓶颈还在,问题没彻底解决,至少一方变化才有可能解决。如果要提高可靠性,确保应用处于健康状态,应用负载要再降低, 牺牲时间换取可靠性

    Last modified: 27 June 2025