注册 登录  
 加关注
   显示下一条  |  关闭
温馨提示!由于新浪微博认证机制调整,您的新浪微博帐号绑定已过期,请重新绑定!立即重新绑定新浪微博》  |  关闭

红烧鱼

linux & windows management

 
 
 

日志

 
 
关于我
mac

专注IT基础架构设计与运维。 欢迎给我留言,或邮件沟通zjwsk@163.com

网易考拉推荐

HP-UX Kernel Configuration for Oracle DBs[zt]   

2008-07-09 23:59:39|  分类: HPUX |  标签: |举报 |字号 订阅

  下载LOFTER 我的照片书  |

Last Update:               May 2004 

Oracle release:     Oracle 10g

Oracle 9i R1/9i R2

Oracle 8/8i

Operating System:     HP-UX 11i v1 (also called 11.11PA-Risc), HP-UX 11.0 (PA-Risc),

HP-UX 11i v2 and v1.6 (Itanium)

Author:          Bernd Menth/HP Oracle CTC

 

Oracle 10g recommendations:

-         Minimum RAM = 512 MB

-         HP-UX 11i v1 (PA-Risc) or v2 (Itanium)

-         Swap Space: Minimum GB or 2 x size of RAM  whichever is greater,
please check Appendix 5 for large systems

-         available Disk Space: 3.5 GB (HP-UX 11.i v1)

this includes 1 GB from companion CD

-          Temporary Disk Space: 400 MB of space in the /tmp directory
(This directory must be writable by the user account performing the installation.)

-         Disk space for DB files: 1 .2 GB

-         Create OS UNIX groups for Oracle system:

o       OSDBA group : Required

o       OSOPER group: Optional

o       ORAINVENTORY group: Optional

o         APACHE group: Recommended

-          Please check appendix 6 for SCHED_NOAGE usage and MLOCK

 

Oracle 9i recommendations:

-         Minimum RAM = 256 MB (HP-UX 11.0/11.i v1)

-         Minimum RAM = 512 MB (HP-UX 11.i v2 and v1.6)

-         HP-UX 11.0 or HP-UX 11i v1 or HP-UX 110 v2.0 and v1.6

-         Swap Space: Minimum 2 x RAM or min  400MB, whichever is greater,
please check Appendix 5 for large systems

-         available Disk Space: 3 GB (HP-UX 11.0/11.i v1)

-         available Disk Space: 3.5 GB (HP-UX 11.i v2 and v1.6)

-         an additional 1 GB for a seed database

-          Temporary Disk Space: up to 2GB of space in the /tmp directory
(If there is not 2 GB of available space in the /tmp directory,
then you can set the TMPDIR or TMP (used by Oracle) environment
variable to point to a directory that has at least 400 MB of available space.
This directory must be writable by the user account performing the installation.)

-         Create OS UNIX groups for Oracle system:

o       OSDBA group : Required

o       OSOPER group: Optional

o       ORAINVENTORY group: Optional

o         APACHE group: Recommended

-          Please check appendix 6 for SCHED_NOAGE usage and MLOCK

 

Oracle requires some patches on the specific Operating System

These patches might be superseded.

HP-UX 11i v1.6 Itanium (based on Oracle release notes Feb 2003):

PHSS_27284, PHSS_27285, PHSS_27286, PHSS_27287, PHSS_27288, PHSS_27289, PHSS_27290, PHSS_27291, PHSS_27292,PHSS_27293, PHKL_28465

If using MC/ServiceGuard A.11.14.01, install OS Patch PHSS_27557

 

HP-UX 11i v1 PA-Risc

PHCO_29287 and PHCO_24402 to relink Oracle

 

HP-UX 11iv1 PA-Risc with Oracle 10g

PHCO_28123, PHKL_29198, PHNE_28476, PHNE_28871, PHSS_28871, PHSS_28880, PHCO_26331, PHCO_29109, PHKL_25468, PHKL_25842, PHKL_25993, PHKL_25994, PHKL_25995, PHKL_26468, PHKL_28489

 

HP provides patch bundles at:      http://www.software.hp.com/SUPPORT_PLUS

HP provides individual patches at:      http://www.itresourcecenter.hp.com

To determine which operating system patches are installed, enter the following command:

$ /usr/sbin/swlist -l patch

 

To determine if a specific operating system patch has been installed, enter the following command:

$ /usr/sbin/swlist -l patch patch_number

 

To determine which operating system bundles are installed, enter the following command:

$ /usr/sbin/swlist -l bundle

 

Required Executables

The make, ar, ld, and nm executables must be present in the /usr/ccs/bin directory. Requires Ansi c cc which is located in /opt/ansic/bin.

 

Additional SW:

To support Oracle Real Application Clusters, the following operating system package and patch are required:

MC/ServiceGuard A.11.14.01

MC/ServiceGuard extension for RAC A.11.14.01

Patch PHSS_27557

 

Note: These are minimum kernel requirements for Oracle9i. If you have previously

tuned your kernel parameters to levels equal to or higher than these values, continue to use the higher values, except for file system cache parameters (dbc_max_pct, dbc_min_pct). The default parameter for files system cache is to high (dbc_max_pct = 50, means 50% of your memory is given to file system cache), it is better to give Oracle the memory to cache data in the SGA instead to have a large files system cache.  A system restart is necessary for kernel changes to take effect.

 

This document include:

 

-      Kernel Parameter

-         Helpful Tools

-         Appendix 1: db_block_size

-         Appendix 2: Raw Devices + Asynchronous IO

-         Appendix 3: Parallel Query Option

-         Appendix 4: shmmax

-         Appendix 5: Swapping/Paging

-         Appendix 6: Privileges for SCHED_NOAGE (for Oracle 9i) and MLOCK

 


 

Kernel Parameter Setting Purpose

 

HP-UX 11.x  

Kernel Parameter

Description, Default/Recommended Value

Bufpages

Number of static Buffer Pages of 4kB,

enables dynamic buffer cache                            (see dbc_max/min_pct).

Not used anymore >= HP-UX 11.23

default= 0

recommended: 0

 

dbc_max_pct

Max dynamic Buffer Cache size in percent

of system memory, when bufpages and

nbuf are both set to 0.

Do not set this value to high, it will use this percentage of the memory to file system cache and give it not to Oracle or any other application!!!!

default=50.

recommended: between 3 and 10 % of the memory

It's more efficient to use system memory

for Oracle SGA instead of file system

buffer cache

 

dbc_min_pct

Min dynamic Buffer Cache size in percent

of system memory (see dbc_max_pct),

default=5.

recommended: between 2 and 5

 

fs_async

see appendix 2 and appendix 6

 

KSI_ALLOC_MAX  

          

 

(NPROC * 8)    

Defines the system wide limit of queued signal that can be allocated.

max_async_ports

Maximum number of asyncdsk ports that can be open at one time. If an error happens with this, it will not shown in the Oracle alert.log  file or in trace files.

default: 50

recommended: max. no of shadow processes + no of parallel querie slaves (could go up to nproc)

max_thread_proc

Max no. of threads allowed in each process

default: 256

recommended:  <= Oracle 8.1.6 = 64

>= Oracle 8.1.7 = 256

Maxfiles

Soft File Limit per Process, soft limit for number of files a process is allowed to have open simultaneously. Processes can increase their soft limit until they reach the hard limit maxfiles_lim.

default=2048

recommended=2048

 

Maxfiles_lim

Hard File Limit per Process,

default=4096

recommended=4096 (default).

 

Maxdsiz

Refers to the maximum data segment size for 32-bit systems. Setting this value too low may cause the processes to run out of memory.

Max Data Segment Size (Bytes)

default: 1073741824

recommended:

<= Oracle 8.1.7 = 256 MB

>= Oracle 9i      = 1073741824 bytes (1 GB)
                             (0
x40000000)

 

Maxdsiz_64bit

Refers to the maximum data segment size for 64-bit systems. Setting this value too low may cause the processes to run out of memory.

Max Data Segment Size (Bytes)

default: 1073741824

 

recommended:

<= Oracle 8.1.7 = 1073741824 bytes (1 GB)
                              (0
x40000000)

>= Oracle 9i       >= 2147483648 bytes (2 GB)
                              (0
x80000000)

 

MAXSSIZ

Defines the maximum stack segment size in bytes for 32-bit systems.

 

Recommended:

>= Oracle 9i  = 134217728 bytes (128 MB)

                          (0x8000000)

 

MAXSSIZ_64BIT

Defines the maximum stack segment size in bytes for 64-bit systems.

 

Recommended:

>= Oracle 9i = 1073741824 bytes (1 GB)

                         (0x40000000)

 


 

Maxswapchunks or swchunk

maximum number of swap chunks  where SWCHUNK is the swap chunk size (1 KB blocks).

(related to parameter swchunk)

 

used with HP-UX 11, 11i, 11iv1.6

not used >= HP-UX 11iv2

 

Default: 2048

recommended:

<= Oracle 8i  = 4096
>= Oracle 9i = 16384

 


 

Maxtsize

Max Text Segment size (Bytes)

Recommended: 134217728 bytes (128 MB)

                          (0x8000000)

not used >= Oracle 10g

max_fcp_reqs

Maximum Number of Concurrent Fiber Channel Requests Per Adapter

recommended: 512

>= HP-UX 11i v1.6 not used anymore

 

Maxuprc

Max Number of simultaneous user                      processes (per user-id!).

default=75

Because all database processes often run

with the Oracle user-id, it's

recommended:

<= Oracle 8.1.7 = maxusers * 5

>= Oracle 9i =  ((NPROC*9)/10)

 

Maxusers

Value of MAXUSERS macro, limits the

Ssstem resource allocation (not the actual

number of users). Influences nproc, ninode, nfile.

default=32.

Not used anymore >= HP-UX 11.23

recommended: at least

 set to number of concurrent Oracle DB users + 64

not used >= Oracle 10g

 

Maxvgs

Max. number of volume groups:

Default: 10

Recommended: increase to the number of volume groups you would like to have on the system (maximum 256)

MSGMAP

Defines the maximum number of message map entries.

Recommended:

>= Oracle 9i = (MSGMNI + 2)

MSGMNI

Defines the number of message queue

identifiers.

Recommended:

>= Oracle 9i = (NPROC)  (at least 4096)

MSGSEG

Defines the number of segments available for messages.

Recommended:

>= Oracle 9i = (NPROC * 4) (at least 32767)

 

MSGTQL

Defines the number of message headers.

Recommended:

>= Oracle 9i = (NPROC) (at least 4096)

 

NCALLOUT

Defines the maximum number of pending timeouts.

Not used anymore >= HP-UX 11.23

Recommended:

>= Oracle 9i = (NPROC + 16)

not used >= Oracle 10g

 

NCSIZE

Defines the Directory Name Lookup Cache (DNLC) space needed for inodes.

Recommended:

>= Oracle 9i = ((NINODE + VX_NCSIZE)

VX_NCSIZE is by default 1024.

 

For >= HP-UX 11iv2 please use = (NINODE + 1024)

 

 

Nfile

Max Number of simultaneously Open

files system-wide at any given time. Total

number of slots it the file descriptor table,

default=16*(nproc+16+maxusers)/10+32+ 2*(npty+nstrpty)

recommended:

<= Oracle 8.1.7 =  to use default.

>= Oracle 9i = (15 * NPROC + 2048)

 

For Oracle installations with a high number of data files this might be not enough, than use the
>= ((number of oracle processes) * (number of Oracle data files) + 2048)

 


 

Nflocks

Max Number of File Locks available

system-wide, (check with glance if you are getting close to the value you have set)

default=200  or  200+10*(num_clients)

recommended:

<= Oracle 8.1.7 = at least 200 + sum off all db_files for all instances of Oracle

>= Oracle 9i = (nproc) (at least 4096)

 

NINODE

Defines the maximum number of open inodes.

Recommended:

>= Oracle 9i = (8 * NPROC + 2048)

 

NKTHREAD

Defines the maximum number of kernel threads supported by the system.

Recommended:

>= Oracle 9i = (((NPROC * 7) / 4) + 16)

 

Nproc

Max Number of Processes that can exist

simultaneously in the system,

default=(20+8*MAXUSERS),

influences ninode, nfile.

recommended:

<= Oracle 8.1.7 = to use default

>= Oracle 9i = 4096

 

Npty

Number of ptys (pseudo ttys),

default=60.

recommended: if you are using rlogin/telnet

connections from clients to server then

increase npty up to the number of client

users, otherwise use default.

Note: ptys are not used when using Oracle SQL*Net

 

num_tachyon_adapters

HP-UX 11.0 only

Number of Tachyon-based Fiber Channel Adapters in the System

Recommended: minimum 5, set to real number of Fiber Channel Adapters

 

HP-UX 11i – no support for tachyon adapters, the new is tachlite, which do not use this kernel parameter.

 


 

o_sync_is_o_dsync

Enable/Disable translation of O_SYNC to

O_DSYNC in open()/fcntl() calls

Not used >= HP-Ux 11iv2

default=0.

recommended:

< Oracle 7.3 = o_sync_is_o_dsync=1 or

use Oracle Patch for Bug #310042.

>= Oracle7.3 = use default for  o_sync_is_o_dsync=0 (patch #310042 not needed).

 

SEMMAP

Defines the maximum number of

semaphore map entries.

Recommended:

>= Oracle 9i = (SEMMNI + 2)

 

Semmni

Number of Semaphore Identifiers,

specifies the maximum number of sets of

semaphores that can exist simultaneously

on the system,

default=64.

recommended:

<= Oracle8i = use default or at least 10

                        per Oracle database

>= Oracle9i = 4096 (HP-UX 11i)

>= Oracle9i = (nproc * 2) at least 4096 for >=HP-UX 11i v1.6

 

 

Semmns

Max Number of Semaphores, defines the

system-wide maximum number of

individual semaphores that can be

allocated for users,

default=128.

recommended:

<= Oracle 8.1.7 = 256 or at least 1 per

                        Oracle process

>= Oracle 9i = (semmni * 2) (HP-UX 11i)

>= Oracle9i = (semmni  * 2) for >= HP-UX 11i v1.6

 

SEMMNU

Defines the number of semaphore undo

structures.

Recommended:

>= Oracle 9i = (NPROC – 4)


SEMVMX

Maximum value of a semaphore.

Recommended:

>= Oracle 9i = 32767

 

shmmax (see appendix 4)

maximum allowable size of one shared memory

segment. The SHMMAX setting should be large

enough to hold the entire SGA in one shared memory segment. A low setting can cause creation of multiple shared memory segments which may lead to performance degradation, but normally it is ok to use a small number of shared memory segments.

 

default=0x04000000 (64MB) Oracle 7

             0x40000000 (1GB =  1073741824 Bytes) 

             Oracle 8, 8i, 9i, 10g

recommended:

db_block_size * db_block_buffers

+ shared_pool_size

+ log_buffer

+ shared memory for the application

or

hold the entire SGA in one shared memory segment !

f.e. 0x100000000 = 4GB

(for 64bit use up to size of physical memory,

 but Oracle can also use multiple segments (please check appendix 4))

 

Shmmni

maximum number of shared memory segments in the entire system.

default=200.

 recommended:

<= Oracle 8.1.7 = use default.

>= Oracle 9i = 512

 

Shmseg

maximum number of shared memory segments one process can attach.

default=12.

recommended:

<= Oracle 8.1.7 = 10 per Oracle database

= Oracle 9i = 32 per Oracle database

>= Oracle10g = 120

 


 

Streampipes

Set this parameter to 0 during on the installation of Oracle DB will reduce the installation time.

Default: 0

Recommended for installation: 0

swapmem_on

(see appendix 5)

Allow Memory to Exceed Swap Space

default: 1 (on)

recommended: 1 (on)

 

Swchunk

Swap Chunk Size (1K Blocks)

(related to kernel parameter maxswapchunks)

default: 2048

recommended: 4096 (up to 65536 for large RAM)

 

VPS_CEILING

maximum System-Selected Page Size in kilobytes.

Default: 64

Recommended:

>= Oracle9i =  64 (up to 16384 = 16MB for large SGA)

 

Helpful Tools:

 

To find out which server you have

-         model

 

To list all kernel parameters on the system

-         use SAM or

-         /usr/sbim/kmtune –l | more

 

To update a kernel parameter

-         Use SAM or

-         /usr/sbin/kmtune –s <kernel parameter> {=|+}

 

To find out 32 or 64 bit HP-UX 11.x

/bin/getconf  KERNEL_BITS

 

Limitations on processes

-         ulimit –a

 

The command sysdef analyzes the currently running system and reports on its tunable configuration parameters.

-         sysdef | more

 

 

To find out  32 or 64 bit Oracle software version

   $ cd $ORACLE_HOME/bin

   $ file oracle

   # if this is the answer, you have 32-bit Oracle

   oracle:  PA-RISC1.1 shared executable dynamically linked -not stripped

   # if this is the answer, you have 64 bit Oracle

   oracle:  ELF-64 executable object file - PA-RISC 2.0 (LP64) # for PA-Risc systems

   oracle:  ELF-64 executable object file - IA64                          #  for Itanium systems


Appendix 1: db_block_size

 

The default db_block_size on HP-UX is 2048. But we recommend to use the following values:

-         db_block_size = 4096 to 8192 for OLTP Applications

-         db_block_size = 8192 to 16384 for DSS/DW Applications

 

For RAC/OPS it makes still sense to use small db_block_size to reduce the amount of data to transfer between the nodes for cache fusion.

 

 

 

 

Appendix 2: Raw Devices + Asynchronous IO

 

We recommend to use raw devices instead of file system files for Oracle

datafiles and to use asynchronous IO, both resulting in much better

performance.

 

Use stripping with a large number of disks to spread IO over multiple disks (high number is good). Define the number of disks in your stripe set also dependent of your future growth of your storage, otherwise you will get probably hot spots in the future. For the stripping are 2 possibility exists. For a high number of disks we did not figured out big difference in performance.

Use the HP-UX Logical Volume Manager to stripe logical volumes across multiple disks. Recommended stripe size: Quite often is 64KB still used

or

use HP-UX Extend-based-striping with 1 MB or even larger extends.

 

To implement asynchronous IO on HP-UX systems (raw devices are required)

-    add the asyncdsk Driver (Asynchronous Disk Pseudo Driver)
      to the HP-UX Kernel (using SAM) and regenerate new kernel

-         create the device file:

-        /sbin/mknod /dev/async c 101 0x0

-        chown oracle:dba /dev/async

-         chmod 660 /dev/async

-         depending on the HP-UX version you have to provide privilege to group dba

- HP-UX 11.0 (dependend on the patch level)
Patch PHKL_22380 and all newer patches which supersede it (like PHKL_22440) use a special "feature" which has to be set to really use asyncio. As root:

/usr/sbin/setprivgrp dba MLOCK

 

- HP-UX 11i (>=11.11)

To use asyncio: As root:

/usr/sbin/setprivgrp dba MLOCK

 

-         set oracle init.ora parameters:

-         use_async_io=true (Oracle 7.3)

-         disk_async_io = true (Oracle 8.0, 8i)

-         db_writers=1 ( < Oracle 7.3)

 

To test the async IO

       - check with        fuser /dev/async

- or look with glance/gpm at the ora_dbwr.

- look for open files:  

- /dev/async   must be shown as opened file

       - if /dev/async is not seen and you want to use asyncio:

To check the priviliges to use async IO on HP-UX 11 (with Patch PHKL_22380 and all newer patches which supersede it like PHKL_22440)  and HP-UX 11i

- To check the privilege capabilities for a group, issue the command:          /usr/bin/getprivgrp
- If the output of getprivgrp(1) does not indicate that the group has the MLOCK privilege, it can be set by issuing the following command as root:
/usr/bin/setprivgrp dba MLOCK

 

When not using asynchronous IO

-         set db_writers to number of disk (for Oracle 7)

-         set db_writer_processes = 1 – 10 (Oracle8.0, 8i)

-         set dbwr_io_slaves = 0 – 999 (Oracle 8.0, 8i)
(Please test with your application, if the behaviour using multiple db_writer_processes is better than use multiple dbwr_io_slaves)

 

Please check also appendix 6 for Oracle 9i (SCHED_NOAGE)

 

 

Appendix 3: Parallel Query Option

 

When using Oracle Parallel Query Option, there are up to parallel_max_servers more Oracle Processes running on the  system. It's  necessary to increase the related HP-UX Parameters  (maxusers, maxuprc, nfile, nproc)

 

 

Appendix 4: shmmax

 

If you got an Oracle error saying ORA-3113: end-of-file on communication channel

than you have to adapt your shmmax to a usable value which means,

-         for Oracle 32-bit max 1GB

-         for Oracle 64-bit shmmax can be raised above 1 GB

 

Please remind, that you have normally more than one shared memory segments (shmmni (max number of shared memory segments allowed to exist simultaneously), shmseg (per process) are the related kernel parameters), which can be used by Oracle. If you have multiple Oracle instances on one system, it is normally better to decrease the size of shared memory and use multiple shared memory segments instead (which is done automatically).

 

Limitations:

-         Oracle7

-         the buffer cache part can be composed of more than one segment,
but the variable portion of the SGA (mainly the shared_pool) must fit in one shared memory segment

-         Oracle8 / Oracle 9i

-         the buffer cache part can be composed of more than one segment,
the variable portion of the SGA (mainly the shared_pool) can be composed of more than one shared memory segment, assuming that the shared memory segments are contiguous

 

Appendix 5: Swapping/Paging

 

-         How much memory does the maschine have?

-         dmesg | grep Physical

-         or grep Physical /var/adm/syslog/syslog.log

 

-         What is the page size?

-         dmesg | grep physical

-         or grep physical /var/adm/syslog/syslog.log

 

-         Is the maschine swapping in the moment?

-         vmstat 5 5

-         page.pi indicated the number of pages moving from disk to memory

-         page.po indicated the number of pages moving from memory to disk

-         memory free indicates the amount of free memory in pages

 

-         How much swap space is configured / used?

-         swapinfo –a

-         A swap area of TYPE=dev gives the NAME of the area of disk used for
paging. /etc/fstab will show NAME as a swap device.

-         A swap area of TYPE=reserve indicates the amount of swap space
that would be required if HP-UX needed to page out all in-memory
pages for all processes that are currently running.

-         A swap area of TYPE=memory only appears if 'memory paging' is enabled,
by setting the configurable kernel parameter swapmem_on to 1 (on).
This default value for this parameter is 1.
'memory paging' is also known as 'pseudo-swap'.
If present, the AVAIL figure is (total memory - memory used after bootup).
This swap area is not actually used by HP-UX for paging (since it does not
make sense to page from memory, and then back into memory).


 

-         How much swap should I configure?

 

-         Oracle 9i

o       Recommendation is to have 2 x RAM size for swap at least 400 MB on a system running Oracle,
but on large systems 2*RAM size is not used anymore. You need at least the swap space that Oracle SGA fits in plus some additional swap space, even swapping out the SGA is not a good solution for a system.
(related kernel parameters for the swap are swchunk and
maxswapchunks)

 

-         Oracle 10g

-         Recommendation is to have at least  1GB or 2 x size of  RAM size for swap on a system running Oracle,
but on large systems with 2 GB or more  2* size of RAM size is not used anymore. You need at least the swap space that Oracle SGA fits in plus some additional swap space, even swapping out the SGA is not a good solution for a system.
(related kernel parameters for the swap are swchunk and
maxswapchunks)

 

 

Appendix 6: Privileges for SCHED_NOAGE (for Oracle 9i) and MLOCK

 

The SCHED_NOAGE policy gives processes holding a latch a fixed priority and makes them nonpreemptable during this time. This causes less latch waits and latch sleeps, hence higher throughput.

To enable the SCHED_NOAGE policy for Oracle, the following init.ora parameter needs to be added:

hpux_sched_noage=154 (for HP/UX 11.0)

hpux_sched_noage=178 (for HP/UX 11i)

In addition, the system privileges RTPRIO and RTSCHED need to be added to the DBA group as root in

/etc/privgroup:

dba RTPRIO RTSCHED

 

 

Complete root user set-up tasks:

The MLOCK privilege grants Oracle the ability to execute asynch I/Os through the HP asynch driver. Without this privilege, Oracle9i generates trace files with the following error message: Ioctl

ASYNCH_CONFIG error, errno = 1

As root, do the following:

_ If it does not already exist, create the /etc/privgroup file. Add the following line to the file:

dba MLOCK RTSCHED RTPRIO

_ Use the following command syntax to assign these privileges:

#setprivgrp groupname privileges

In the preceding command, groupname is the name of the group that receives the privileges, and privileges are the privileges that are granted to the group.

For example, to set the privileges for the OSDBA group, which is typically named dba, enter the following command:

#setprivgrp dba MLOCK RTSCHED RTPRIO

  评论这张
 
阅读(1044)| 评论(0)
推荐 转载

历史上的今天

在LOFTER的更多文章

评论

<#--最新日志,群博日志--> <#--推荐日志--> <#--引用记录--> <#--博主推荐--> <#--随机阅读--> <#--首页推荐--> <#--历史上的今天--> <#--被推荐日志--> <#--上一篇,下一篇--> <#-- 热度 --> <#-- 网易新闻广告 --> <#--右边模块结构--> <#--评论模块结构--> <#--引用模块结构--> <#--博主发起的投票-->
 
 
 
 
 
 
 
 
 
 
 
 
 
 

页脚

网易公司版权所有 ©1997-2017