Golang在Ubuntu 14.04 LTS中大量读取tcp ip:port I/O超时

问题描述

我编写了一个golang程序，该程序在过去几个月中在ubuntu 12.04 LTS中运行良好，直到我将其升级到14.04 LTS

I wrote a golang program which run well in the past several months in ubuntu 12.04 LTS until I upgraded it to 14.04 LTS

我的程序专注于发送HTTP请求，该请求每秒发送大约2-10个HTTP请求.HTTP请求地址各不相同.

My program is focused on sending HTTP requests which send about 2-10 HTTP requests per second. The HTTP request address vary.

发生问题时，首先，某些请求显示 read tcp [ip]:[port]:I/o超时，然后几分钟后，所有请求都显示 read tcp [ip]:[port]:I/O超时，无法发送任何请求.

When the problem occurs, first, some of the requests shows read tcp [ip]:[port]: i/o timeout, then after several minutes all requests show read tcp [ip]:[port]: i/o timeout, not any request can be sent.

我重新启动程序，一切恢复正常.

我们所有的服务器(2台服务器)从12.04升级到14.04后都存在此类问题

我为每个请求创建新的goroutine

问题不会在相同的时间间隔内发生，有时不会在一两天内发生，有时会在一小时内发生两次

以下是我请求HTTP地址的代码:

Bellow is my code requesting HTTP Address:

t := &http.Transport{
    Dial:            timeoutDial(data.Timeout),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
//req := s.ParseReq(data)
req := data.convert2Request()
if req == nil {
    return
}

var resp *http.Response
if data.Redirect {
    c := &http.Client{
        Transport: t,
    }
    resp, err = c.Do(req)
} else {
    resp, err = t.RoundTrip(req)
}

data.updateTry()

r := s.ParseResp(data, resp, err)

updateTry:

updateTry:

func (d *SendData) updateTry() {
    d.Try++
    d.LastSend = time.Now()
}

timeoutDial:

timeoutDial:

func timeoutDial(timeout int) func(netw, addr string) (net.Conn, error) {
    if timeout <= 0 {
        timeout = 10
    }
    return func(netw, addr string) (net.Conn, error) {
        deadline := time.Now().Add(time.Duration(timeout) * time.Second)
        c, err := net.DialTimeout(netw, addr, time.Second*time.Duration(timeout+5))
        if err != nil {
            return nil, err
        }
        c.SetDeadline(deadline)
        return c, nil
    }
}

我处理的回复是:

func (s *Sender) ParseResp(data SendData, resp *http.Response, err error) (r Resp) {
    r = Resp{URL: data.URL}
    if err != nil {
        r.Err = err.Error()
    } else {
        r.HttpCode = resp.StatusCode
        r.Header = resp.Header
        r.URL = resp.Request.URL.String()
        defer resp.Body.Close()
        // we just read part of response and log it.
        reader := bufio.NewReader(resp.Body)
        buf := make([]byte, bytes.MinRead) // 512 byte
        for len(r.Body) < 1024 {           // max 1k
            var n int
            if n, _ = reader.Read(buf); n == 0 {
                break
            }
            r.Body += string(buf[:n])
        }
    }
    return
}

我还发现/etc/sysctl.conf中的设置可以使问题发生的频率降低:

I also found setting in /etc/sysctl.conf which can make the problem happen less frequently:

net.core.somaxconn = 65535
net.netfilter.nf_conntrack_max = 655350
net.netfilter.nf_conntrack_tcp_timeout_established = 1200

我需要帮助来解决这个问题.

I need help for solving this problem.

似乎是这样，但我看不到任何解决方案 https://bugs.launchpad.net/juju-core/+ bug/1307434

It seems like this but I don't see any solution https://bugs.launchpad.net/juju-core/+bug/1307434

推荐答案

要更明确地声明Not_a_Golfer和OneOfOne所说的内容，当您完成响应后，需要关闭保持打开状态的连接(通过Body字段，即io.ReadCloser).因此，基本上，一个简单的方法是将与发出http请求有关的代码更改为:

To more explicitly state what Not_a_Golfer and OneOfOne have said, when you're done with the response, you need to close the connection which has been left open (through the Body field which is an io.ReadCloser). So basically, one simple though would be to change the code pertaining to making an http request to:

var resp *http.Response
if data.Redirect {
    c := &http.Client{
        Transport: t,
    }
    resp, err = c.Do(req)
} else {
    resp, err = t.RoundTrip(req)
}
if err == nil {
    defer resp.Body.Close() // we need to close the connection
}

这篇关于Golang在Ubuntu 14.04 LTS中大量读取tcp ip:port I/O超时的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！