Nginx 反向代理 Varnish做网站缓存加速器_freeke

1、Freeke 2008-12-10Nginx 反向代理+Varnish 技术实现目标：一、 Nginx 反向代理1、 安装条件：Nginx: http:/sysoev.ru/nginx/nginx-0.6.32.tar.gzSSL: /source/openssl-0.9.8g.tar.gzPcre: ftp:/ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.7.tar.gzZlib: 2、 安装：l Ssl安装：rootChinarenservice oracle# tar -zxvf o

2、penssl-0.9.8g.tar.gzrootChinarenservice oracle# cd openssl-0.9.8grootChinarenservice openssl-0.9.8g# ./config -prefix=/usr/local/openssl/rootChinarenservice openssl-0.9.8g# makerootChinarenservice openssl-0.9.8g# make installl Pcre 安装：rootChinarenservice oracle# tar -zxvf pcre-7.7.tar.gzrootChinaren

3、service oracle# cd pcre-7.7rootChinarenservice pcre-7.7# ./configure -prefix=/usr/local/pcrerootChinarenservice pcre-7.7# makerootChinarenservice pcre-7.7# make installMake 时报错：libtool: ignoring unknown tag CXXlibtool: unrecognized option -DHAVE_CONFIG_H'Try libtool -help' for more informati

4、on.make1: * pcrecpp.lo Error 1make1: Leaving directory /home/beijing/pcre-7.7'make: * all Error 2原因：pcre-7.7 configuration summary: Install prefix . : /usr/local/pcre C preprocessor . : gcc -E C compiler . : gcc C+ preprocessor . : C+ compiler . : Linker . : /usr/bin/ld C preprocessor flags . :

5、C compiler flags . : -O2 C+ compiler flags . : Linker flags . : Extra libraries . :没有装GCC C+包：gcc-c+-3.4.6-8.i386.rpm libstdc+-devel-3.4.6-8.i386.rpml Zlib 安装：rootChinarenservice oracle# tar -zxvf zlib-1.2.3.tar.gzrootChinarenservice oracle# cd zlib-1.2.3rootChinarenservice zlib-1.2.3# ./configure -

6、prefix=/usr/local/zlibrootChinarenservice zlib-1.2.3# makerootChinarenservice zlib-1.2.3# make installl Tcmalloc 安装：rootchinarenservice # wget /releases/libunwind/libunwind-0.99-alpha.tar.gzrootchinarenservice # tar zxvf libunwind-0.99-alpha.tar.gzrootchinarenservice #

7、 cd libunwind-0.99-alpha/rootchinarenservice # CFLAGS=-fPIC ./configurerootchinarenservice # make CFLAGS=-fPICrootchinarenservice # make CFLAGS=-fPIC installrootchinarenservice # wget http:/google-rootchinarenservice # tar zxvf google-perftools-0.98.tar.gzrootchinarenservice # cd google-perftools-0.

8、98/rootchinarenservice # ./configurerootchinarenservice # make && make installrootchinarenservice # echo "/usr/local/lib" > /etc/ld.so.conf.d/usr_local_lib.confrootchinarenservice # ldconfigroot chinarenservice local# lsof -n | grep tcmallocnginx 7323 root mem REG 8,2 1412859 44

9、0730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7324 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7325 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7326 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7327 www mem REG 8,2 14

10、12859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7328 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7329 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7330 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7331 www mem RE

11、G 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0l Nginx 安装：rootChinarenservice oracle# tar -zxvf nginx-0.6.32.tar.gzrootChinarenservice oracle# cd nginx-0.6.32rootChinarenservice nginx-0.6.32# ./configure -prefix=/usr/local/nginx -with-http_ssl_module -with-pcre=/root/pcre-7.7 -with-zlib=/ro

12、ot/zlib-1.2.3 -with-http_stub_status_module -with-http_realip_module -with-http_addition_module -with-http_sub_module -with-http_dav_module -with-http_flv_module -with-openssl=/root/openssl-0.9.8g -with-google_perftools_modulerootChinarenservice nginx-0.6.32# makerootChinarenservice nginx-0.6.32# ma

13、ke install./configure -with-http_ssl_module -with-pcre=./pcre-7.8 -with-zlib=./zlib-1.2.3 -with-http_stub_status_module -with-openssl=./openssl-0.9.8g3、 配置：rootChinarenservice oracle# cat /usr/local/nginx/conf/nginx.confuser nobody nobody;worker_processes 30;error_log logs/error.log notice;pid logs/

14、nginx.pid;events use epoll; worker_connections 40960;http include mime.types;default_type application/octet-stream;log_format main '$remote_addr - $remote_user $time_local ' '"$request" $status $bytes_sent ' '"$http_referer" "$http_user_agent" '

15、'"$gzip_ratio"'keepalive_timeout 150;server_names_hash_bucket_size 64;upstream cache ip_hash;server 66:8080; /varnish server 1server ; /apache server server listen :80;server_name ;access_log logs/.access.log main;location / proxy_pass http:/cache;

16、proxy_redirect /; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Is-EDU 0;client_max_body_size 50m;                &#

17、160;  client_body_buffer_size 256k;                   proxy_connect_timeout 10;                   proxy_send_timeou

18、t 15;                   proxy_read_timeout 15;                   proxy_buffer_size 4k;     

19、0;             proxy_buffers 4 32k;                   proxy_busy_buffers_size 64k;           &#

20、160;       proxy_temp_file_write_size 64k; server listen :81;server_name ;location /NginxStatus stub_status on;access_log off;allow 47;allow;deny all; 以上配置为nginx 做反向代理，监听:80的IP，接收 的域名请求，转发到后端varnish缓存服务器4、 优化：l 修改open

21、files数显示open files数rootChinarenservice oracle# ulimit -acore file size (blocks, -c) 0data seg size (kbytes, -d) unlimitedfile size (blocks, -f) unlimitedpending signals (-i) 1024max locked memory (kbytes, -l) 32max memory size (kbytes, -m) unlimitedopen files (-n) 1024修改open files数rootChinarenservic

22、e oracle# ulimit -n 8192l 优化Linux内核参数rootChinarenservice oracle# vi /etc/sysctl.conf在末尾增加以下内容：net.ipv4.tcp_fin_timeout = 30net.ipv4.tcp_keepalive_time = 300net.ipv4.tcp_syncookies = 1net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1net.ipv4.ip_local_port_range = 5000 65000使配置立即生效：rootChinarenserv

23、ice oracle# /sbin/sysctl -pnet.ipv4.tcp_fin_timeout = 30#表示如果套接字由本端要求关闭，这个参数决定了它保持在FIN-WAIT-2状态的时间。net.ipv4.tcp_keepalive_time = 300#表示当Keepalive起用的时候，TCP发送keepalive消息的频繁度。默认值是2小时，这里我改为5分钟。net.ipv4.tcp_syncookies = 1#表示开启SYN Cookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，预设为0，表示关闭，我将它开启。net.ipv4.tcp_

24、tw_reuse = 1#表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，预设为0，表示关闭，我将它开启。net.ipv4.tcp_tw_recycle = 1#表示开启TCP连接中TIME-WAIT sockets的快速回收，预设为0，表示关闭，我将它开启。net.ipv4.ip_local_port_range = 5000 65000#表示用来向外连接的端口范围。预设情况下很小：32768到61000，我将它改为5000到65000。net.ipv4.tcp_max_syn_backlog = 8192#表示SYN队列的长度，预设为1024，加大队列长度为8

25、192，可容纳更多等待连接的网络连接数。net.ipv4.tcp_max_tw_buckets = 5000#表示系统同时保持TIME_WAIT套接字的最大数量，如果超过这个数字，TIME_WAIT套接字将立刻被清除并印出警告讯息。预设是180000，改为5000。之后记得执行 /sbin/sysctl -p 使配置生效。l 不停止Nginx服务的情况下平滑变更Nginx配置rootChinarenservice oracle# kill -HUP cat /usr/local/nginx/logs/nginx.pidl5、 日志处理由于在Nginx上代理的网站访问量非常大，每天的日志达到1G

26、以上，如果不对日志分天处理，那日志文件将非常大，并且不方便进行日志分析。l Logrotate/home/log/*access.log daily missingok rotate 31 nocompress prerotate #/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=shutter endscript postrotate if -f /usr/local/nginx/logs/nginx.pid ; then kill -USR1 cat /usr/local/nginx/logs/nginx.pid

27、fi endscript l Shell写一个shell，每天在晚上11点59分执行将日志文件分天处理：rootChinarenservice log# crontab l59 23 * * * /var/InfiNET/usemon/logcronnginx.shrootChinarenservice usemon# less logcronnginx.sh#!/bin/sh/bin/mv /home/log/bbs.access.log /home/log/bbs.access.date +%Y%m%d.log/bin/mv /home/log/register.access.log /h

28、ome/log/register.access.date +%Y%m%d.log/bin/mv /home/log/shutter.access.log /home/log/shutter.access.date +%Y%m%d.log#/bin/mv /home/log/bbs.access.log /home/log/bbs.access.%YYYY-24-%MM-24-%DD-24.log#/bin/mv /home/log/register.access.log /home/log/register.access.%YYYY-24-%MM-24-%DD-24.log#/bin/mv /

29、home/log/shutter.access.log /home/log/shutter.access.%YYYY-24-%MM-24-%DD-24.log/usr/bin/killall -s USR1 nginxl6、 awstats日志分析请参见Freeke Nginx Awstats Install7、二、 Varnish 缓存Varnish优点：1、Varnish采用了“Visual Page Cache”技术，在内存的利用上，Varnish比Squid具有优势，它避免了Squid频繁在内存、磁盘中交换文件，性能要比Squid高。2、Varnish的稳定性非常好3、通过Varnis

30、h管理端口，可以使用正则表达式快速、批量地清除部分缓存，这一点是Squid不能具备的。Varnish网站缓存加速器安装：1、创建www用户和组，以及Varnish缓存文件存放目录（/var/InfiNET/cache）：rootChinarenservice oracle# /usr/sbin/groupadd www -g 48rootChinarenservice oracle# /usr/sbin/useradd -u 48 -g www wwwrootChinarenservice oracle# mkdir -p /var/InfiNET/cacherootChinarenservi

31、ce oracle# chmod +w /var/InfiNET/cacherootChinarenservice oracle# chown -R www:www /var/InfiNET/cache2、创建Varnish日志目录（/var/logs/）：rootChinarenservice oracle# mkdir -p /usr/local/varnish/logsrootChinarenservice oracle# chmod +w /usr/local/varnish/logsrootChinarenservice oracle# chown -R www:www /usr/l

32、ocal/varnish/logs3、编译安装varnish：下载：rootChinarenservice oracle# wget rootChinarenservice oracle# tar zxvf varnish-1.1.2.tar.gzrootChinarenservice oracle# cd varnish-1.1.2rootChinarenservice oracle# ./configure -prefix=/usr/local/varnishrootChinarenservice oracle# make && make install./configur

33、e -enable-debugging-symbols -enable-developer-warnings -enable-dependency-tracking -prefix=/usr/local/varnish注意,我在进行make步骤时,出现如下错误: "varnishhist.c:35:20: error: curses.h: No such file or directory" 造成该问题的原因是因为系统中少了ncurses-devel包4、创建Varnish配置文件：rootChinarenservice oracle# vi /usr/local/varn

34、ish/vcl.confbackend mblog set backend.host = "68" set backend.port = "80" acl purge "localhost" "" ""/24; ""/24;sub vcl_recv if (req.http.User-Agent "(Baiduspider|google)")   

35、0;                     error 405 "Not allowed."                 if (req.http.Referer "(|2

36、|7878758.com|||||7878758.com||||||||72|4)")                      &

37、#160;  error 405 "Not allowed."        if (req.request = "PURGE") if (!client.ip purge) error 405 "Not allowed." lookup; if (req.http.host "") set req.backend = mblog; if (req.request != "GET" && req.re

38、quest != "HEAD") pipe; else lookup; else error 404 "Koncept Cache Server" lookup; sub vcl_hit if (req.request = "PURGE") set obj.ttl = 0s; error 200 "Purged." sub vcl_miss if (req.request = "PURGE") error 404 "Not in cache." sub vcl_fetch i

39、f (req.request = "GET" && req.url ".(txt|js|gif|jpg|jpeg|tom|swf|css)$") set obj.ttl = 3600s; else set obj.ttl = 30d; 对以上配置文件解释一下：(1)、Varnish通过反向代理请求后端IP为，端口为80的apache服务器；(2)、Varnish允许localhost、、10.167.0.*源IP通过PURGE方法清除缓存；(3)、Varnish对域名为的请求进行处理，非域名的请求则

40、返回“freeke Cache Server”；(4)、Varnish对HTTP协议中的GET、HEAD请求进行缓存，对POST请求透过，让其直接访问后端Web服务器。之所以这样配置，是因为POST请求一般是发送数据给服务器的，需要服务器接收、处理，所以不缓存；(5)、Varnish对以.txt和.js等结尾的URL缓存时间设置1小时，对其它的URL缓存时间设置为30天。5、启动VarnishrootChinarenservice oracle# limit -SHn 51200rootChinarenservice oracle# /usr/local/varnish/sbin/varnis

41、hd -n /var/InfiNET/cache -f /usr/local/varnish/vcl.conf -a :8080 -s file,/var/InfiNET/cache/varnish_cache.data,2G -g www -u www -w 30000,51200,10 -T :10030 -p client_http11=on -P /usr/local/varnish/var/varnish.pid6、启动varnishncsa用来将Varnish访问日志写入日志文件：rootChinarenservice oracle# /usr/lo

42、cal/varnish/bin/varnishncsa -n /var/InfiNET/cache -w /usr/local/varnish/logs/varnish.log &7、配置开机自动启动VarnishrootChinarenservice oracle# vi /etc/rc.localulimit -SHn 51200/usr/local/varnish/sbin/varnishd -n /var/InfiNET/cache -f /usr/local/varnish/vcl.conf -a :8080 -s file,/var/InfiNET/cache

43、/varnish_cache.data,2G -g www -u www -w 30000,51200,10 -T :10030 -p client_http11=on -P /usr/local/varnish/var/varnish.pid/usr/local/varnish/bin/varnishncsa -n /var/InfiNET/cache -w /usr/local/varnish/logs/varnish.log&8、优化Linux内核参数rootChinarenservice oracle# vi /etc/sysctl.confnet.ipv4.

44、tcp_fin_timeout = 30net.ipv4.tcp_keepalive_time = 300net.ipv4.tcp_syncookies = 1net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1net.ipv4.ip_local_port_range = 5000    65000再看看如何管理Varnish：1、查看Varnish服务器连接数与命中率：rootChinarenservice oracle# /usr/local/varnish/bin/varnishstat n /var/InfiNET

45、/cache/2、通过Varnish管理端口进行管理：用help看看可以使用哪些Varnish命令：rootChinarenservice oracle# /usr/local/varnish/bin/varnishadm -T :3500 helpAvailable commands:ping timestampstatusstartstopstatsvcl.load vcl.inline vcl.use vcl.discard vcl.listvcl.show param.show -l param.set help commandurl.purge dump.pool3

46、、通过Varnish管理端口，使用正则表达式批量清除缓存：(1)、例：清除类似rootChinarenservice oracle# /usr/local/varnish/bin/varnishadm -T :3500 url.purge /a/(2)、例：清除类似rootChinarenservice oracle# /usr/local/varnish/bin/varnishadm -T :3500 url.purge w*$(3)、例：清除所有缓存：rootChinarenservice oracle# /usr/local/varnish/bin/v

47、arnishadm -T :3500 url.purge *$4、一个清除Squid缓存的PHP函数（清除Varnish缓存同样可以使用该函数，无需作任何修改，十分方便）：view plaincopy to clipboardprint?1. <?php   2. function purge($ip, $url)   3.   4.     $errstr = ''   5.  

48、60;  $errno = ''   6.     $fp = fsockopen ($ip, 80, $errno, $errstr, 2);   7.     if (!$fp)   8.        9.     &

49、#160;    return false;   10.        11.     else  12.        13.         $out = "PURGE $url HTTP/1.1rn" &

50、#160; 14.         $out .= "Host:rn"   15.         $out .= "Connection: closernrn"   16.         fputs ($f

51、p, $out);   17.         $out = fgets($fp , 4096);   18.         fclose ($fp);   19.         return true;  

52、0;20.        21.   22.   23. purge("", "/index.php");   24. ?>  附1：Varnish官方网站/附2：日志处理2007年12月10日，我写了一个每天0点运行，按天切割Varnish日志，生成一个压缩文件，同时删除上个月旧日志的脚本（/usr/local/varnish

53、/logs/cutlog.sh）：rootChinarenservice oracle# /usr/local/varnish/logs/cutlog.sh #!/bin/sh# This file run at 00:00date=$(date -d "yesterday" +"%Y-%m-%d")pkill -9 varnishncsamv /usr/local/varnish/logs/youvideo.log /usr/local/varnish/logs/$date.log/usr/local/varnish/bin/varnishncsa -

54、n /var/InfiNET/cache -w /usr/local/varnish/logs/varnish.log &mkdir -p /usr/local/varnish/logs/youvideo/gzip -c /usr/local/varnish/logs/$date.log > /usr/local/varnish/logs/youvideo/$date.log.gzrm -f /usr/local/varnish/logs/$date.logrm -f /usr/local/varnish/logs/youvideo/$(date -d "-1 mont

55、h" +"%Y-%m*").log.gz设置在每天00:00定时执行： rootChinarenservice oracle# /usr/bin/crontab -e或者 rootChinarenservice oracle# vi /var/spool/cron/root0 0 * * * /bin/sh /usr/local/varnish/logs/cutlog.sh附3: varnish日志的rotaterootChinarenservice oracle# touch /etc/logrotate.d/varnishrootChinarenservice oracle# les /etc/logrotate.d/varnish/usr/local/varnish/logs/varnish.log     daily     rotate 60    &