Nginx Connection 不夠用 的參數調整

機器 Loading 是 "0.x", 但是連線總是會出現找不到資料, 重新、再試一次的訊息, 如下述:

HTTP request sent, awaiting response... No data received.
Retrying.

直覺猜測就是 Connection、檔案數、TCP... 等滿了的問題, 於是作些調整.

註: netstat -ant | grep TIME_WAIT | wc -l # TIME_WAIT 數值應該很高

Nginx worker_connections 不夠用 的系統參數調整

在此直接把調整的參數寫出來(環境: Ubuntu Linux 12.04), 需要系統這些設定, 否則 Nginx worker_connections 無法調整到 1024 以上.

  1. vim /etc/sysctl.conf
    net.core.netdev_max_backlog = 65536
    net.core.rmem_max = 16777216
    net.core.somaxconn = 65536
    net.core.wmem_max = 16777216
    net.ipv4.tcp_max_syn_backlog = 65536
    net.ipv4.tcp_moderate_rcvbuf = 1
    net.ipv4.tcp_no_metrics_save = 1
    net.ipv4.tcp_rfc1337 = 1
    net.ipv4.tcp_rmem = 4096 87380 16777216
    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_wmem = 4096 87380 16777216
    vm.mmap_min_addr = 0
    vm.swappiness = 0

    net.core.netdev_max_backlog = 65536
    net.core.rmem_max = 16777216
    net.core.somaxconn = 65536
    net.core.wmem_max = 16777216
    net.ipv4.tcp_congestion_control = bic
    net.ipv4.tcp_ecn = 1
    net.ipv4.tcp_frto = 0
    net.ipv4.tcp_max_syn_backlog = 1024
    net.ipv4.tcp_moderate_rcvbuf = 1
    net.ipv4.tcp_no_metrics_save = 0
    net.ipv4.tcp_rfc1337 = 1
    net.ipv4.tcp_rmem = 4096 87380 4194304
    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_syncookies = 1
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_tw_recycle = 0
    net.ipv4.tcp_tw_reuse = 0
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_wmem = 4096 87380 4194304
    vm.mmap_min_addr = 4096
    vm.swappiness = 0

    或 (下述取自: Ubuntu 高並發狀態下的linux內核網路參數調優)

    net.ipv4.tcp_fin_timeout = 30
    net.ipv4.tcp_keepalive_time = 1200
    net.ipv4.route.gc_timeout = 100
    net.ipv4.ip_local_port_range = 1024 65000
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_syn_retries = 1
    net.ipv4.tcp_synack_retries = 1
    net.ipv4.tcp_max_syn_backlog = 262144
    net.core.netdev_max_backlog = 262144
    net.core.somaxconn = 262144
    net.ipv4.tcp_mem = 94500000 915000000 927000000

    或 (下述取自: Linux下高並發socket最大連接數所受的各種限制) - 推薦此設定

    net.ipv4.ip_local_port_range = 1024 65536
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.tcp_rmem = 4096 87380 16777216
    net.ipv4.tcp_wmem = 4096 65536 16777216
    net.ipv4.tcp_fin_timeout = 10
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_timestamps = 0
    net.ipv4.tcp_window_scaling = 0
    net.ipv4.tcp_sack = 0
    net.core.netdev_max_backlog = 30000
    net.ipv4.tcp_no_metrics_save = 1
    net.core.somaxconn = 262144
    net.ipv4.tcp_syncookies = 0
    net.ipv4.tcp_max_orphans = 262144
    net.ipv4.tcp_max_syn_backlog = 262144
    net.ipv4.tcp_synack_retries = 2
    net.ipv4.tcp_syn_retries = 2

    或 目前我使用的版本

    net.ipv4.ip_local_port_range = 1024 65535
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.tcp_rmem = 4096 87380 16777216
    net.ipv4.tcp_wmem = 4096 65536 16777216
    net.ipv4.tcp_fin_timeout = 10
    # net.ipv4.tcp_tw_recycle = 1
    # net.ipv4.tcp_timestamps = 0 # recycle 等同無效
    net.ipv4.tcp_window_scaling = 0
    net.ipv4.tcp_sack = 0
    net.core.netdev_max_backlog = 30000
    net.ipv4.tcp_no_metrics_save = 1
    net.core.somaxconn = 65535
    net.ipv4.tcp_syncookies = 0
    net.ipv4.tcp_max_orphans = 262144
    net.ipv4.tcp_max_syn_backlog = 262144
    net.ipv4.tcp_synack_retries = 2
    net.ipv4.tcp_syn_retries = 2
    
    # 此參數適情況加入,限制 TIME_WAIT 數量
    # net.ipv4.tcp_tw_reuse = 1
    # net.ipv4.tcp_keepalive_time = 1200
    # 強制將 TIME_WAIT 限制在這個數字,Debian 預設 131072
    # net.ipv4.tcp_max_tw_buckets = 32768
    # net.ipv4.tcp_max_tw_buckets = 65535
    
    # modprobe ip_conntrack
    # 遇到 kernel nf_conntrack: table full, dropping packet 加入
    net.ipv4.netfilter.ip_conntrack_max = 655350
    net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 1200
  2. $ sysctl -p # sysctl.conf 設定重新載入
    • 若出現此錯誤: error: "Invalid argument" setting key "net.ipv4.ip_local_port_range"
    • 此行改成: net.ipv4.ip_local_port_range = 1024 65535 # 註: port 1024 以下系統用, 65536 以上設定會失敗, 所以設小一點
    • 若出現: setting key "net.core.somaxconn": 不適用的引數
    • 可以設成: net.core.somaxconn = 65535 # 一樣是超過 65535 就會不吃的參數.
  3. $ sysctl -w net.ipv4.route.flush=1 # 此行可以不執行.
  4. vim /etc/security/limits.conf
    # 於檔案最後新增下述 (ulimit -n 655360)
    * soft nofile 655360
    * hard nofile 655360
  5. vim /etc/nginx/nginx.conf
    worker_rlimit_nofile 102400;
    events {
        worker_connections 1024; # 可以加到 4096
        #worker_connections 768;
        use epoll;
        # multi_accept on;
    }
    http {
        #keepalive_timeout 65;
        keepalive_timeout 1;
        ....
    }
  6. 註: worker_connections 1024, worker_processes 4 的狀況下, 理論值的最大連線數 = worker_processes * worker_connections / 4

相關網頁

Save

作者: Tsung

對新奇的事物都很有興趣, 喜歡簡單的東西, 過簡單的生活.

在〈Nginx Connection 不夠用 的參數調整〉中有 1 則留言

發表迴響

這個網站採用 Akismet 服務減少垃圾留言。進一步了解 Akismet 如何處理網站訪客的留言資料