AIX性能调优PPT学习教案

1、会计学1AIX性能调优性能调优第1页/共90页Performance Analysis ToolsCPUMemoryI/O SubsysNetwork Subsvmstat, iostatvmstatiostat, vmstatlsattrpslspslspsnetstatsarsvmonlsattrnfsstatgprof/prof/tprof filemonlsdevnetpmontime/timexlspv,lslv,lsvgifconfignetpmonfileplaceiptrace/ipreportlocktracefilemontcpdumpemstat, alstatlvmst

2、attopastopastopastopasPTX/xmperfPTX/xmperfPTX/xmperfPTX/xmperftrace, trcrpttrace, trcrpttrace, trcrpttrace, trcrptcurt, splat,trusstrusstrusstrussprocmon,lparstat, mpstat,cpupstatprocmon,lparstatnfs4clPerformance Tuning Toolsnice/renicevmo/vmtunevmo/vmtunenoschedo/schedtuneioo/vmtuneioo/vmtunenfsobi

3、ndprocessorchps, mkpschdevchdevchdevfdprmigratepvifconfigsetprichdevchlvbindintcpurmssreorgvgprocmon AIX Performance Tools第2页/共90页Scheduling Policies?SCHED_FIFO?Non-preemptive scheduling?Fixed priority?SCHED_RR?Round-Robin scheduling?Fixed priority?Runs until interrupted or end of time slice?SCHED_O

4、THER?Same as SCHED_RR but applies to threads with non-fixed priorities?Priorities degrade with CPU Usage第3页/共90页Scheduling Policies (continued)?Additional Scheduling Policies (all at fixed priority) ?SCHED_FIFO2SCHED_FIFO2?Thread is put at head of its run queue if it was only asleep for a short peri

5、od of time (within affinity_lim ticks)?Can be preempted by a higher priority thread?SCHED_FIFO3SCHED_FIFO3?Thread is always put at head of its run queue whenever it becomes runnable?Can be preempted by a high priority thread第4页/共90页Process and Thread PriorityProcess PrioritiesHighLow040255Real-Time

6、Process PrioritiesUser Process Prioritiessetpri()nice, reniceParentProcessChildProcess/threadsfork()InheritedProcess PriorityPRI = p_nice + CPU_PENALTYp_nice = 40 + NICEps -el (to view values)ps -emo THREAD (to view threads)第5页/共90页Scheduling Formula priority = 40 +priority = 40 + NICENICE + + xnice

7、xnice + ( + (CPU_Penalty CPU_Penalty * * X X) )If NICE = 20, then xnice = 0If NICE 20, then If NICE 20, then xnicexnice = NICE - = NICE - 2020 CPU_Penalty CPU_Penalty = = recent CPU usagerecent CPU usage * * r r ( (r r = = R R/32)/32) X X = (40 + = (40 + NICENICE + + xnicexnice + 4) / 64 (1, if NICE

8、 = 20) + 4) / 64 (1, if NICE = 20) CPU usageCPU usage is decayed is decayed once per secondonce per second at a factor at a factor d d CPU usageCPU usage = CPU usage = CPU usage * * d d ( (d d = D/32) = D/32)第6页/共90页CPU Monitoring Tools vmstat, sar, ps# sar -u 5 3(CPU)19:28:02%usr%sys%wio%idle19:28:

9、070609419:28:121409519:28:0774090Average34093# sar -c 1 3(Syscalls)19:28:25scall/ssread/sswrit/sfork/sexec/srchar/swchar/s19:28:261343610.000.002691306151719:28:27463410.000.002716922153119:28:28463410.000.0027169221531Average753510.000.0027083291527# sar -q 5 319:31:42runq-sz %runoccswpq-sz%swpocc1

10、9:31:471.01001.010019:31:522.01001.010019:31:571.01001.0100Average1.3951.095第7页/共90页CPU Monitoring Tools vmstat, sar, ps第8页/共90页Locating Dominant Processes Current Data - what processes are currently using the most CPU? Current Data - what processes are currently using the most CPU? ps auxps aux ps

11、vxps vx ps -eF “%c %p %C”|sort -rn +2ps -eF “%c %p %C”|sort -rn +2tprof -x sleep 60 tprof -x sleep 60 (_prof.all)(_prof.all)Show ThreadsShow Threadsps -e-k -mo THREAD -p ps -e-k -mo THREAD -p Show Overall Status Show Overall Status topastopas第9页/共90页第10页/共90页第11页/共90页第12页/共90页第13页/共90页第14页/共90页第15页/

12、共90页第16页/共90页Application Tuning Techniques?Commands for Tuning?nice?renice?schedo ex. schedo o sched_R = 30?API:?setpri()?thread_setsched()?If CPU bound:?Profiling (prof, gprof, tprof)?Tracing第17页/共90页?What is a thread model ?What is a thread model ?A user thread has to be mapped to a kernel thread

13、to be executedA user thread has to be mapped to a kernel thread to be executed?The way this mapping is done is called the thread model The way this mapping is done is called the thread model ?Three possible thread modelsThree possible thread models?M:1 model M:1 model the pthreads lib schedules the

14、user threadsthe pthreads lib schedules the user threads?1:1 model1:1 model?M:N model M:N model export AIXTHREAD_SCOPE=Sexport AIXTHREAD_SCOPE=PThread Models第18页/共90页VMM Memory SegmentsRealMemoryFrameRemote FSNFSClientPersistent SegmentFileLocalPageFile SystemPaging SpaceFile systemProcess codePaging

15、 spaceProcess private, dataRemote files and executablesNFS, JFS2CDROM FSClient SegmentWorking Segment第19页/共90页Computational vs File MemoryComputational Memory:File MemoryPagesfromPersistentorClientWorking Storage SegmentProgram Text SegmentEverything ElseWorking第20页/共90页Page Replacement BasicsPage r

16、eplacement is the process of free pages in real memory by scanning the Page Frame Table (PFT) for pages to stealPage replacement starts ifPage replacement starts ifThe number of free memory pages gets low (The number of free memory pages gets low (minfreeminfree threshold)threshold)The number of cac

17、hed file pages gets close to its strict limit The number of cached file pages gets close to its strict limit ( (maxclient maxclient for client pages and for client pages and maxpermmaxperm for JFS pages if for JFS pages if strict_maxpermstrict_maxperm is enabled) is enabled)第21页/共90页Page Replacement

18、Real AddrSeg TypeREFMODaaa1W+aaa3Wbbb1P+bbb3Pccc1C+ccc3CPage Frame Table (excerpt)Real AddrSeg TypeREFMODaaa1W+aaa2W+aaa3W+aaa4Wbbb1P+bbb2P+bbb3P+bbb4Pccc1C+ccc2C+ccc3C+ccc4CPaging SpacePersistentPFT (excerpt)NFS ServerReal Addr.aaa2aaa4bbb2bbb4ccc2ccc4Free ListResulting PFT (excerpt)Second Chance:L

19、eft in memory,but REF bit cleared第22页/共90页Memory Pools ?Real memory is divided into one or more Memory PoolsMemory Pools?The number of memory pools depends on the amount of real memory and the number of CPUs ?Each memory poolEach memory pool has its own page replacement daemon, lrudlrud?Memory pools

20、 are divided into one or more FramesetsFramesets which allows concurrent memory allocation out of a memory pool?Memory allocation is done round-robin across the memory pools where two pages per pool are allocated before moving to the next ? vmstat -v | grep memory poolsvmstat -v | grep memory pools?

21、vmo -r -o mempools=nbr_of_memory_pools vmo -r -o mempools=nbr_of_memory_pools (require bosboot and reboot) ?vmo r o framesets=nbr_of_framesets vmo r o framesets=nbr_of_framesets (e.g. 2)?vmo o lrubucket=new_value vmo o lrubucket=new_value (default is 131072 frames per bucket)第23页/共90页Address space o

22、f 32-bit processes16 segments of 256 MB each01432KernelProgram Text Program Data 15141312Shared Library Text Shared Library DataKext, mem mapping.Memory Mapping:shmget()shmat()SegmentOffset428第24页/共90页Large Page Support?AIX supports virtual pages size 4KB ,64KB,16 MB,16GB?Can improve the performance

23、 of some application due to reduced TLB (Translation look aside buffer) misses ?Large pages are pinnedpinned at system boot?Large page support can be enabled/disabled with vmovmo?The page size can be changed to 16MB and the number of large page regions can be set based on the applications needs?Usin

24、g large pages may degrade performance in some cases:?particularaly for applications that fork/exec often第25页/共90页Using Large Page Support?Use the ld ld -blpdata-blpdata option at to enable LP for data and heap?Use ldedit ldedit command to set or clear large page support on an executable?ldedit -blpd

25、ata ldedit -blpdata to set?ldedit -bnolpdata ldedit -bnolpdata to clear?Large page support can be set through LDR_CNTRL, environment variable?LDR_CNTRL=LARGE_PAGE_DATA=Y | N | M (mandatory)第26页/共90页Tools vmstat, psvmstat -1 5vmstat -I 1 5vmstat s (for paging and system events since initialization)vm

26、stat f (# of forks since initialization)vmstat v ps aux, ps ev, ps evx (5.3 only) 第27页/共90页svmon Tool?Captures a snapshot of the current state of memory, Captures a snapshot of the current state of memory, and then analyze it using four different reportsand then analyze it using four different repor

27、ts Global svmon -G Global svmon -G mem leak tool Process svmon -P Process svmon -P mem leak tool Segment svmon -S Segment svmon -S Detailed segment svmon -D Detailed segment svmon -D svmon Put 3svmon Put 3 svmon P pid; ps v pid svmon P pid; ps v pid mem leak tool RSS=seg.1+2+15 (no 13), TRS=seg.1 TS

28、IZ第28页/共90页Tuning File Memory Usage?Real memory made up of computational pages and file pages?VMM parameters minperm and maxperm specify memory thresholds?If percentage of RAM occupied by file pages falls below minperm, VMM steals both file and computational pages?If percentage of RAM occupied by fi

29、le pages rises above maxperm, VMM steals only file pages?If percentage of RAM occupied by file pages is between minperm and maxperm, VMM will steal only file pages unless file repaging rate is higher than computational repaging rate第29页/共90页Page Replacement ParametersVMM tunable Function 5.1 tuning

30、5.2/5.3 tuningminfreeSpecifies the minimum number of frames on the free list at which the VMM starts to steal pages to replenish the free listvmtune -f vmo -o minfreemaxfreethe number of frames on the free list at which page-stealing is to stopvmtune -Fvmo -o maxfreeminpermSpecifies the point below

31、which the page stealer will steal file or computational pages regardless of repaging ratesvmtune -p vmo -o minperm%maxpermSpecifies the point above which the page steal algorithm steals only file pagesvmtune -Pvmo -o maxperm%第30页/共90页Page Replacement ParametersVMM tunable Function 5.1 tuning 5.2/5.3

32、 tuningstrict_maxpermSets a hardlimit on persisent file cacheingvmtune -hvmo -o strict_maxpermmaxclientSpecifies the point above which page-stealing-algorithm steals only client pagesvmtune -tvmo -o maxclient%lrubucketSpecifies the number of pages per LRU bucketvmtune -lvmo -o lrubucketframesetsSpec

33、ifies the number of framesets per memory poolvmtune -v vmo -o framesetsmempoolsSpecifies the number of memory poolsvmtune -mvmo -o mempoolsvmstat v to look at results (minfree/maxfree adjustment)第31页/共90页Deferred Page Space Allocation?On some systems, page space may not ever be needed even if all th

34、e pages accessed have been touched. This is most common on systems with very large amount of RAM.?Deferred Page Space Allocation policy will further delay allocation of paging space until it is necessary to page out the pageuntil it is necessary to page out the page - which results in no wasted pagi

35、ng space allocation. This can save huge amounts of paging space.?This may result in overcommitment of paging space in cases where more virtual memory than available RAM is accessed.?vmo -o defps = vmo -o defps = will turn off/on D.P.S.A.?Npswarn and npskill to determine low paging space condition?SI

36、GDANGER and SIGKILL第32页/共90页?Paging space supports working set ?Database raw logical volume situation?Tuning minperm and maxperm?ps au command?Advanced tool: svmon?rmss tool?Use rmss to find out minimum memory requirementTips for Application Tuning第33页/共90页Storage Hierarchymy file/homeFILESYSTEMFILE

37、DIRECTORY/home/myfileLOGICAL PARTITIONSPHYSICALVOLUMEPHYSICALPARTITIONSLOGICAL VOLUMEhd1hdisk0hdisk1VOLUMEGROUP第34页/共90页LVM PoliciesINTER-POLICYMaximum Inter-Disk Policy (Range=maximum) with a Single Logical Volume Copy per Disk (Strict=y)INTRA-POLICYhdxPartition 2hdxPartition1 (Copy)hdxPartition 1h

38、dxPartition 2(Copy)Inner EdgeInner MiddleCenterOuter MiddleOuter Edge第35页/共90页LVM PoliciesINTRA-POLICY Inner Edge内缘内缘Inner MiddleCenterOuter MiddleOuter EdgeProblem disk? 第36页/共90页Logical Volume StripingStripe Unit 1Stripe Unit 4Stripe Unit nStripe Unit n+3Stripe Unit 1Stripe Unit 4Stripe Unit nStri

39、pe Unit n+3Stripe Unit 2Stripe Unit 5Stripe Unit n+1Stripe Unit n+4Stripe Unit 3Stripe Unit 6Stripe Unit n+2Stripe Unit n+5First StripeSecond StripeFirst PhysicalVolumeSecond PhysicalVolumeThird PhysicalVolume第37页/共90页I/O PathFilesystem I/O application applicationapplicationRaw LVM I/ORaw Disk I/O F

40、ilesystem VMM Direct I/OLogical volumeLogical volume adapter adapter disk disk disk adapter/dev/hdisk2 blk mode/dev/rdisk2 char mode第38页/共90页Logical Volume Fragmentation?To view policies?lslv or smit lslv?Inter/intra-policies, PP size and others?To check fragmentation?lslv -l lvname?shows COPIES and

41、 IN BAND information?To see placement on hd?lslv -p hdisk# lvname?To view regions?lspv -p hdisk#第39页/共90页Sample Output - LV Commands# lslv lv04LOGICAL VOLUME:lv04LV IDENTIFIER: 00000145315bbdd4.2VG STATE:active/completeTYPE:jfsMAX LPs:128COPIES:1LPs:62STALE PPs:0INTER-POLICY: minimumINTRA-POLICY: mi

42、ddleMOUNT POINT: /tstcMIRROR WRITE CONSISTENCY: onEACH LP COPY ON A SEPARATE PV ?: yesVOLUME GROUP:datavgPERMISSION:read/writeLV STATE:opened/syncdWRITE VERIFY: offPP SIZE:4 megabyte (s)SCHED POLICY:parallelPPs:62BB POLICY:relocatableRELOCATABLE:yesUPPER BOUND:32LABEL:/tstc# lslv -l lv04lv04: /tstcP

43、VCOPIESIN BANDDISTRIBUTIONhdisk1062:000:00059%000:037:000:025:000第40页/共90页Logical Volume Performance?What we can do : ?Allocate Hot LVs to Different PVs?Spread Hot LV Across Multiple PVs?Place Hottes LVs in Center of PVs, except for LVs which have Mirror Write Consistency Check turned on?Define LV t

44、o Maximum Size that you will need?reorgvg volgrp lvname lvname第41页/共90页For Highest LVM Performance?Number of copies of each LP: 1 (no mirroring)?If mirroring is needed, set:?Scheduling policy: parallel?Allocation policy is each copy on separate PV?Write verify: No?MWC: off?Inter policy: maximum (R/W

45、 shared among PVs)第42页/共90页JFS/JFS2 Comparisons第43页/共90页File System Organization (JFS)1第44页/共90页File System I/O Layers第45页/共90页File System Performance Measurements第46页/共90页Determining Fragmentation using fileplacefilemon o filename.out -O all ; sleep 20 ; trcstopfileplace -ipv /unix第47页/共90页第48页/共90

46、页?Writes to a file will be synchronous if the file was opened with O_SYNC or O_DSYNC open flag?Each write will be flushed to the disk before the write returnsSynchronous I/Odiskwrite()schedule write write return 第49页/共90页Asynchronous I/O第50页/共90页?No tuning necessary for Async I/O using fastpath (I/O

47、 to raw LVM No tuning necessary for Async I/O using fastpath (I/O to raw LVM files for example)files for example)?When not using fastpath, queuing is done and aioserver kprocs read When not using fastpath, queuing is done and aioserver kprocs read requests from the AIO queuesrequests from the AIO qu

48、eues?SMITs AIO configuration panel has several parameters that can be SMITs AIO configuration panel has several parameters that can be tunedtuned?maxreqs - max number of AIOs that can be queuedmaxreqs - max number of AIOs that can be queued?maxservers - max number of AIO server kprocs that can be ma

49、xservers - max number of AIO server kprocs that can be created (per CPU since AIX 5.2)created (per CPU since AIX 5.2)?minserver - minimum number of AIO serverkprocs that will minserver - minimum number of AIO serverkprocs that will be created when AIO is configuredbe created when AIO is configuredTu

50、ning Asynchronous I/O第51页/共90页?Direct I/O is an alternative non-caching policy which causes file data to be transferred directly to/from the disk to/from the applications user-space buffer without going through the VMM file cache?Applications that do not have high file cache rates can use Direct I/O

51、 to reduce CPU utilization for reads and writes by eliminating the copy from memory cache to the user buffer?Direct I/O can be enabled through the dio mount dio mount optionoption as well as by open flag (O_DIRECT) open flag (O_DIRECT) ?mount -o dio /filesystemDirect I/O第52页/共90页?Performance of Dire

52、ct I/O readsPerformance of Direct I/O reads?Each Direct I/O read causes a synchronous read from disk; this can result in poor performance if the data was likely to be in memory under the normal caching policy.?Direct I/O currently bypasses the VMM read-ahead algorithm which is very effective for seq

53、uential reads of JFS files. Direct I/O readers can compensate for the loss of this read-ahead by issuing larger read requests (minimum of 128K) or by issuing asynchronous direct I/O using multiple threads or by using the Asynchronous I/O subroutines (aio_read, lio_listio).?Performance of Direct I/O

54、writesPerformance of Direct I/O writes?Direct I/O writes cause synchronous writes to disk instead of copying to memory and writing to disk later via the write-behind algorithms or via file syncs. This can cause a performance penalty for files opened with Direct I/O.?Using Direct I/O avoids caching p

55、ages in RAM thus limiting how many pages the syncd daemon has to flush.Performance Costs of Direct I/O第53页/共90页sync/fsync第54页/共90页?AIX uses a special logical volume called the log device as a circular journal for recording modifications to filesystem meta-data.?Filesystem meta-data include the super

56、block, inodes, indirect data pointers, and directories.?When meta-data is modified, a duplicate transaction is made to the JFS log.?When a sync/fsync occurs, commit records are written to the JFS log to indicate that modified pages in memory have been committed to disk.JFS Log第55页/共90页?Create new JF

57、S or JFS2 log logical volume:?smit mklvor mklv -t jfslog -y LVname VGname 1 PVname?For JFS2, smit mklv or mklv -t jfs2log -y LVname VGname 1 PVname?Unmount the filesystem and then format the log?/usr/sbin/logform -V /dev/LVname?Modify /etc/filesystems and LVCB:?chfs -a log=/dev/LVname /filesystemnam

58、e?mount filesystemCreating Additional JFS/JFS2 Logs第56页/共90页?After identifying a fragmented file system:?backup the files (by name) in that filesystem?delete the contents of the filesystem? restore contentions of filesystem?Should not be done for all filesystems (/tmp, /usr, /)Reorganizing the File

59、System第57页/共90页?Write-behind involves write modified pages in memory to disk after reaching a threshold ?Sequential write-behind initiates I/O pages if the VMM detects that writing is sequential?Files are divided into 16KB partitions or 4 pages?If a page in a partition is modified, and if any pages

60、in the previous partition are dirty, then the pages of the previous partition are written to the disk.?Sequential write-behind tunable with vmtune -c /iooJFS Sequential Write-Behind第58页/共90页?Similiar to JFS, JFS2 also has a sequential write behind feature?JFS2 also partitions a file into clusters of