10 月 232014
10 月 232014
日志 前端两台 负载均衡 后端一台 保存和分析
Nginx对已经生成日志进行按天截断
主机A
脚本
#!/bin/bash
logs_path=”/opt/nginx-1.6.2/logs/”
# for bbs.access.log
mv ${logs_path}bbs.access.log ${logs_path}bbs.access.log.$(date -d “yesterday” +”%Y%m%d”).A
# for www.access.log
mv ${logs_path}www.access.log ${logs_path}www.access.log.$(date -d “yesterday” +”%Y%m%d”).A
# for shop.access.log
mv ${logs_path}shop.access.log /mnt/backup/shop.access.log.$(date -d “yesterday” +”%Y%m%d”).A
kill -USR1 `cat /opt/nginx-1.6.2/logs/nginx.pid`
加入定时任务
00 0 * * * root /opt/nginx-1.6.2/logs/logs.sh
主机B
在备份和分析服务主机上对已经截断的前一天日志进行合并
脚本
#!/bin/bash
yesterday=`date -d ‘-1 day’ +%Y%m%d`
# for static.linuxcache.cn
mv /backup/mp/h154.110/static.access.log.$yesterday.A /backup/logs/static.access.log.$yesterday
cat /backup/mp/h154.111/static.access.log.$yesterday.B >>/backup/logs/static.access.log.$yesterday
rm -rf /backup/mp/h154.111/static.access.log.$yesterday.B
加入定时任务 执行日志生成后一段时间
安装EPEL源，安装awstats二进制版本
[root@backup ~]# yum install httpd awstats
Installing:
awstats noarch 7.0-3.el6 epel 1.2 M
Installing for dependencies:
mailcap noarch 2.1.31-2.el6 base 27 k
perl-Compress-Raw-Zlib x86_64 1:2.021-136.el6 base 69 k
perl-Compress-Zlib x86_64 2.021-136.el6 base 45 k
perl-HTML-Parser x86_64 3.64-2.el6 base 109 k
perl-HTML-Tagset noarch 3.20-4.el6 base 17 k
perl-IO-Compress-Base x86_64 2.021-136.el6 base 69 k
perl-IO-Compress-Zlib x86_64 2.021-136.el6 base 135 k
perl-URI noarch 1.40-2.el6 base 117 k
perl-libwww-perl noarch 5.833-2.el6 base 387 k
修改安装后生成的apache配置文件
新增站点配置文件
/usr/share/awstats/tools/awstats_configure.pl
You can then manually update your statistics for ‘bbs.linuxcache.cn’ with command:
> perl awstats.pl -update -config=bbs.linuxcache.cn
You can also build static report pages for ‘bbs.linuxcache.cn’ with command:
> perl awstats.pl -output=pagetype -config=bbs.linuxcache.cn
手动执行日志分析
/usr/share/awstats/wwwroot/cgi-bin/awstats.pl -update -config=bbs.linuxcache.cn
[root@backup awstats]# /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -update -config=bbs.linuxcache.cn
Create/Update database for config “/etc/awstats/awstats.bbs.linuxcache.cn.conf” by AWStats version 7.0 (build 1.971)
From data in log file “/backup/logs/bbs.access.log.20141021″…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Jumped lines in file: 0
Parsed lines in file: 3
Found 0 dropped records,
Found 0 comments,
Found 0 blank records,
Found 0 corrupted records,
Found 0 old records,
Found 3 new qualified records.
[root@backup awstats]#
修改配配置文件
1修改日志路径
2禁用反向解析
3增加地理数据库
4增加UTF8解码器
http://192.168.153.111/awstats/awstats.pl?config=
增加http验证
[root@backup ~]# htpasswd -c /etc/awstats/htpasswd.users logsviewer
New password:
Re-type new password:
Adding password for user logsviewer
New password:
[root@backup ~]#
修改apache配置
<Directory “/usr/share/awstats/wwwroot”>
AuthName “Linuxcache.com LOGS ANALYSIS”
AuthType Basic
AuthUserFile /etc/awstats/htpasswd.users
Require valid-user
Options None
AllowOverride None
Order allow,deny
#Allow from 127.0.0.1
Allow from all
</Directory>
===================================增加地理数据库
[root@backup ~]# rpm -lq GeoIP
/etc/GeoIP.conf
/etc/GeoIP.conf.default
/usr/bin/geoiplookup
/usr/bin/geoiplookup6
/usr/bin/geoipupdate
/usr/lib64/libGeoIP.so.1
/usr/lib64/libGeoIP.so.1.5.1
/usr/lib64/libGeoIPUpdate.so.0
/usr/lib64/libGeoIPUpdate.so.0.0.0
/usr/share/GeoIP
/usr/share/GeoIP/GeoIP.dat
/usr/share/GeoIP/GeoIPASNum.dat
/usr/share/GeoIP/GeoIPASNumv6.dat
/usr/share/GeoIP/GeoIPv6.dat
/usr/share/GeoIP/GeoLiteASNum.dat
/usr/share/GeoIP/GeoLiteASNumv6.dat
/usr/share/GeoIP/GeoLiteCity.dat
/usr/share/GeoIP/GeoLiteCityv6.dat
/usr/share/GeoIP/GeoLiteCountry.dat
/usr/share/doc/GeoIP-1.5.1
/usr/share/doc/GeoIP-1.5.1/AUTHORS
/usr/share/doc/GeoIP-1.5.1/COPYING
/usr/share/doc/GeoIP-1.5.1/ChangeLog
/usr/share/doc/GeoIP-1.5.1/README
/usr/share/doc/GeoIP-1.5.1/TODO
/usr/share/doc/GeoIP-1.5.1/fetch-geoipdata-city.pl
/usr/share/doc/GeoIP-1.5.1/fetch-geoipdata.pl
/usr/share/man/man1/geoiplookup.1.gz
/usr/share/man/man1/geoiplookup6.1.gz
/usr/share/man/man1/geoipupdate.1.gz
[root@backup ~]#
[root@backup ~]# rpm -lq GeoIP-update
/etc/cron.weekly/geoipupdate
[root@backup ~]#
修改配置文件
#LoadPlugin=”geoip GEOIP_STANDARD /pathto/GeoIP.dat”
LoadPlugin=”geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat”
#LoadPlugin=”geoip_city_maxmind GEOIP_STANDARD /pathto/GeoIPCity.dat”
LoadPlugin=”geoip_city_maxmind GEOIP_STANDARD /usr/share/GeoIP/GeoLiteCity.dat”
错误分析
[root@backup awstats]# /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -update -config=static.linuxcache.cn
Error: Plugin load for plugin ‘geoip’ failed with return code: Error:
Can’t locate Geo/IP.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .
/usr/share/awstats/lib /usr/share/awstats/plugins) at (eval 2) line 1.
Can’t locate Geo/IP/PurePerl.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5
/usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .
/usr/share/awstats/lib /usr/share/awstats/plugins) at (eval 3) line 1.
Error: Need Perl module Geo::IP or Geo::IP::PurePerl
Setup (‘/etc/awstats/awstats.static.linuxcache.cn.conf’ file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in ‘docs’ directory).
[root@backup awstats]#
[root@backup awstats]# yum install perl-Geo-IP
安装后执行
[root@backup awstats]# /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -update -config=static.linuxcache.cn
Warning: Tried to load plugin “geoip” twice. Fix config file.
Create/Update database for config “/etc/awstats/awstats.static.linuxcache.cn.conf” by AWStats version 7.0 (build 1.971)
From data in log file “/backup/logs/static.access.log.20141022″…
Phase 1 : First bypass old records, searching new record…
Direct access after last parsed record (after line 6)
Jumped lines in file: 6
Found 6 already parsed records.
Parsed lines in file: 0
Found 0 dropped records,
Found 0 comments,
Found 0 blank records,
Found 0 corrupted records,
Found 0 old records,
Found 0 new qualified records.
[root@backup awstats]#
分析搜索关键词乱码问题
#LoadPlugin=”decodeutfkeys”
LoadPlugin=”decodeutfkeys”
10 月 212014
single
confirm
10 月 212014
memory监控free不出图的问题分析
[root@labs05 ~]# snmpwalk -v 2c -c public 127.0.0.1 .1.3.6.1.4.1.2021.4.6.0
UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 14184408 kB
[root@labs05 ~]# free
total used free shared buffers cached
Mem: 16280408 2095372 14185036 0 139496 1348900
-/+ buffers/cache: 606976 15673432
Swap: 16777208 0 16777208
[root@labs05 ~]# snmpwalk -v 2c -c public 192.168.150.101 .1.3.6.1.4.1.2021.4.6.0
UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 14855716 kB
[root@labs05 ~]#
[root@labs01 ~]# free
total used free shared buffers cached
Mem: 16280408 1424864 14855544 0 128304 984548
-/+ buffers/cache: 312012 15968396
Swap: 16416760 0 16416760
[root@labs01 ~]#
free 内存大于10G的问题 实际14G
10 000 000
100 000 000
[root@labs05 rra]# rrdtool info h_dev150_101_mem_free_159.rrd
filename = “h_dev150_101_mem_free_159.rrd”
rrd_version = “0003”
step = 300
last_update = 1408672203
ds[mem_free].type = “GAUGE”
ds[mem_free].minimal_heartbeat = 600
ds[mem_free].min = 0.0000000000e+00
ds[mem_free].max = 1.0000000000e+07
修改后原图无变化，删除重新生成后，可见变更
[root@labs05 rra]# rrdtool info h_dev150_101_mem_free_163.rrd |grep .max
ERROR: opening ‘h_dev150_101_mem_free_163.rrd’: No such file or directory
[root@labs05 rra]# rrdtool info h_dev150_101_mem_free_163.rrd |grep .max
ds[mem_free].max = 1.0000000000e+08
[root@labs05 rra]#
[root@labs05 rra]# rrdtool info h_dev150_108_mem_free_173.rrd |grep .max
ds[mem_free].max = 1.0000000000e+08
[root@labs05 rra]# rrdtool info h_dev150_108_mem_buffers_171.rrd |grep .max
ds[mem_buffers].max = 1.0000000000e+08
[root@labs05 rra]# rrdtool info h_dev150_108_mem_cache_172.rrd |grep .max
ds[mem_cache].max = 1.0000000000e+08
[root@labs05 rra]#
方法
Console -> Data Templates -> (Edit)
Data Templates [edit: ucd/net – Memory – Buffers]
Data Templates [edit: ucd/net – Memory – Cache]
Data Templates [edit: ucd/net – Memory – Free]
Maximum Value (‘U’ for No Maximum) 10000000
改 100000000
10 月 182014
配置
harveymei@SRX240H2# show snmp community public { authorization read-only; clients { 192.168.153.110/32; } } [edit] harveymei@SRX240H2#
配置命令
10 月 182014
10 月 182014
9 月 282014
^
9 月 272014
设备 IBM V3700
Brocade 300
Qlogic 2560
Dell R720
存储
已划分LUN
主机
已新增主机，主机和LUN映射已作
存储多路径设备需要的软件支持 默认情况下CenOS6.5最小化安装镜像已包含
[root@localhost ~]# rpm -qa |grep device-mapper
device-mapper-libs-1.02.79-8.el6.x86_64
device-mapper-event-libs-1.02.79-8.el6.x86_64
device-mapper-event-1.02.79-8.el6.x86_64
device-mapper-persistent-data-0.2.8-2.el6.x86_64
device-mapper-1.02.79-8.el6.x86_64
device-mapper-multipath-libs-0.4.9-72.el6.x86_64
device-mapper-multipath-0.4.9-72.el6.x86_64
[root@localhost ~]#
系统环境 CenOS 6.5 x86_64 minimal ISO
未安装FC HBA的情况下 multipathd 服务为停止状态 且不含配置文件
[root@localhost ~]# service multipathd status
multipathd is stopped
[root@localhost ~]# chkconfig –list multipathd
multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@localhost ~]#
[root@localhost ~]# lsmod |grep multipath
[root@localhost ~]#
增加最小化配置文件
blacklist {
devnode “^sda”
}
defaults {
user_friendly_names yes
path_grouping_policy multibus
failback immediate
no_path_retry fail
}
加载内核模块和启动服务
# modprobe dm-multipath
# modprobe dm-round-robin
# service multipathd start
# multipath -v2
查看设备状态和路径识别
已安装HBA卡的主机安装系统后，每台主机安装两块HBA卡 QLogic 2560单端口8Gbps
服务自动启动，且包含配置文件
查看默认服务状态和已加载内核模块
[root@node1 ~]# chkconfig –list multipathd
multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@node1 ~]#
[root@node1 ~]# lsmod |grep multipath
dm_multipath 17724 2 dm_round_robin
dm_mod 84209 18 dm_multipath,dm_mirror,dm_log
[root@node1 ~]#
查看默认配置文件
查看多路径设备识别
单机使用FC SAN存储 逻辑卷管理 1物理卷2卷组3逻辑卷4文件系统挂载
9 月 252014
……