Nginx Connection 不夠用 的參數調整

機器 Loading 是 "0.x", 但是連線總是會出現找不到資料, 重新、再試一次的訊息, 如下述:

HTTP request sent, awaiting response... No data received.
Retrying.

直覺猜測就是 Connection、檔案數、TCP... 等滿了的問題, 於是作些調整.

註: netstat -ant | grep TIME_WAIT | wc -l # TIME_WAIT 數值應該很高

Nginx worker_connections 不夠用 的系統參數調整

在此直接把調整的參數寫出來(環境: Ubuntu Linux 12.04), 需要系統這些設定, 否則 Nginx worker_connections 無法調整到 1024 以上.

1. vim /etc/sysctl.conf

   net.core.netdev_max_backlog = 65536
   net.core.rmem_max = 16777216
   net.core.somaxconn = 65536
   net.core.wmem_max = 16777216
   net.ipv4.tcp_max_syn_backlog = 65536
   net.ipv4.tcp_moderate_rcvbuf = 1
   net.ipv4.tcp_no_metrics_save = 1
   net.ipv4.tcp_rfc1337 = 1
   net.ipv4.tcp_rmem = 4096 87380 16777216
   net.ipv4.tcp_sack = 1
   net.ipv4.tcp_timestamps = 1
   net.ipv4.tcp_window_scaling = 1
   net.ipv4.tcp_wmem = 4096 87380 16777216
   vm.mmap_min_addr = 0
   vm.swappiness = 0

   或

   net.core.netdev_max_backlog = 65536
   net.core.rmem_max = 16777216
   net.core.somaxconn = 65536
   net.core.wmem_max = 16777216
   net.ipv4.tcp_congestion_control = bic
   net.ipv4.tcp_ecn = 1
   net.ipv4.tcp_frto = 0
   net.ipv4.tcp_max_syn_backlog = 1024
   net.ipv4.tcp_moderate_rcvbuf = 1
   net.ipv4.tcp_no_metrics_save = 0
   net.ipv4.tcp_rfc1337 = 1
   net.ipv4.tcp_rmem = 4096 87380 4194304
   net.ipv4.tcp_sack = 1
   net.ipv4.tcp_syncookies = 1
   net.ipv4.tcp_timestamps = 1
   net.ipv4.tcp_tw_recycle = 0
   net.ipv4.tcp_tw_reuse = 0
   net.ipv4.tcp_window_scaling = 1
   net.ipv4.tcp_wmem = 4096 87380 4194304
   vm.mmap_min_addr = 4096
   vm.swappiness = 0

   或 (下述取自: Ubuntu 高並發狀態下的linux內核網路參數調優)

   net.ipv4.tcp_fin_timeout = 30
   net.ipv4.tcp_keepalive_time = 1200
   net.ipv4.route.gc_timeout = 100
   net.ipv4.ip_local_port_range = 1024 65000
   net.ipv4.tcp_tw_reuse = 1
   net.ipv4.tcp_tw_recycle = 1
   net.ipv4.tcp_syn_retries = 1
   net.ipv4.tcp_synack_retries = 1
   net.ipv4.tcp_max_syn_backlog = 262144
   net.core.netdev_max_backlog = 262144
   net.core.somaxconn = 262144
   net.ipv4.tcp_mem = 94500000 915000000 927000000

   或 (下述取自: Linux下高並發socket最大連接數所受的各種限制) - 推薦此設定

   net.ipv4.ip_local_port_range = 1024 65536
   net.core.rmem_max = 16777216
   net.core.wmem_max = 16777216
   net.ipv4.tcp_rmem = 4096 87380 16777216
   net.ipv4.tcp_wmem = 4096 65536 16777216
   net.ipv4.tcp_fin_timeout = 10
   net.ipv4.tcp_tw_recycle = 1
   net.ipv4.tcp_timestamps = 0
   net.ipv4.tcp_window_scaling = 0
   net.ipv4.tcp_sack = 0
   net.core.netdev_max_backlog = 30000
   net.ipv4.tcp_no_metrics_save = 1
   net.core.somaxconn = 262144
   net.ipv4.tcp_syncookies = 0
   net.ipv4.tcp_max_orphans = 262144
   net.ipv4.tcp_max_syn_backlog = 262144
   net.ipv4.tcp_synack_retries = 2
   net.ipv4.tcp_syn_retries = 2

   或 目前我使用的版本

   net.ipv4.ip_local_port_range = 1024 65535
   net.core.rmem_max = 16777216
   net.core.wmem_max = 16777216
   net.ipv4.tcp_rmem = 4096 87380 16777216
   net.ipv4.tcp_wmem = 4096 65536 16777216
   net.ipv4.tcp_fin_timeout = 10
   # net.ipv4.tcp_tw_recycle = 1
   # net.ipv4.tcp_timestamps = 0 # recycle 等同無效
   net.ipv4.tcp_window_scaling = 0
   net.ipv4.tcp_sack = 0
   net.core.netdev_max_backlog = 30000
   net.ipv4.tcp_no_metrics_save = 1
   net.core.somaxconn = 65535
   net.ipv4.tcp_syncookies = 0
   net.ipv4.tcp_max_orphans = 262144
   net.ipv4.tcp_max_syn_backlog = 262144
   net.ipv4.tcp_synack_retries = 2
   net.ipv4.tcp_syn_retries = 2
   
   # 此參數適情況加入，限制 TIME_WAIT 數量
   # net.ipv4.tcp_tw_reuse = 1
   # net.ipv4.tcp_keepalive_time = 1200
   # 強制將 TIME_WAIT 限制在這個數字，Debian 預設 131072
   # net.ipv4.tcp_max_tw_buckets = 32768
   # net.ipv4.tcp_max_tw_buckets = 65535
   
   # modprobe ip_conntrack
   # 遇到 kernel nf_conntrack: table full, dropping packet 加入
   net.ipv4.netfilter.ip_conntrack_max = 655350
   net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 1200

2. $ sysctl -p # sysctl.conf 設定重新載入
   • 若出現此錯誤: error: "Invalid argument" setting key "net.ipv4.ip_local_port_range"
   • 此行改成: net.ipv4.ip_local_port_range = 1024 65535 # 註: port 1024 以下系統用, 65536 以上設定會失敗, 所以設小一點
   • 若出現: setting key "net.core.somaxconn": 不適用的引數
   • 可以設成: net.core.somaxconn = 65535 # 一樣是超過 65535 就會不吃的參數.
3. $ sysctl -w net.ipv4.route.flush=1 # 此行可以不執行.
4. vim /etc/security/limits.conf

   # 於檔案最後新增下述 (ulimit -n 655360)
   * soft nofile 655360
   * hard nofile 655360

5. vim /etc/nginx/nginx.conf

   worker_rlimit_nofile 102400;
   events {
       worker_connections 1024; # 可以加到 4096
       #worker_connections 768;
       use epoll;
       # multi_accept on;
   }
   http {
       #keepalive_timeout 65;
       keepalive_timeout 1;
       ....
   }

6. 註: worker_connections 1024, worker_processes 4 的狀況下, 理論值的最大連線數 = worker_processes * worker_connections / 4

相關網頁

• 調整 /etc/sysctl.conf 網路參數提高系統負載
• 啟用 Nginx Status 的設定
• Nginx 出現 500 Error 修復 (too many open file, connection)
• Nginx 中 worker_connections 的問題
• 一個和 worker_connections 參數相關的 Nginx 系統問題及解決辦法
• How To Optimize Nginx For Maximum Performance
• 通過 ulimit 改善系統性能
• ulimit
• 修改 Ubuntu ulimit 限制
• http長連接 200萬嘗試及調優
• nginx + linux優化，提高併發性能
• 調整Linux TCP參數 提升伺服器性能
• Part 1: Lessons learned tuning TCP and Nginx in EC2 « Chartbeat Engineering Blog

Save