Opet problem sa CentOS-om

Pa onda mi to sve vise i vise vuce na hardware... Srdjan ti je dao lep predlog sto se memorije tice...

---

Ja ne razumem zasto produkcijski server NE MOZE u runlevel 3. Pa valjda produkcijski server MORA da bude u runlevel 3, zasto produkcijski server (koji pri tome vrti apache) ima dignute X-e???

I meni sve mirise na hardver sve vise.... Moja ideja: Ako je samo za internu upotrebu - dodji nekim vikendom pa pokreni memtest? Ono subota/nedelja vece.

Bez toga da nadjes neku gresku u logovima i bez toga da hoces da proveris da li ti hardver radi kako treba, ima JAKO malo sanse da ti neko na forumu moze pomoci.

---

ne vidim sta bi moglo tako uporno da zakucava server osim neki hardverski problem, recimo pregrijavanje. cak bi se i neki problem sa memorijom mogao vidjeti u logu (e.g. smrt procesa usljed nedostatka memorije, ECC greske i sl.). za pocetak bih obratio paznju na temperaturu (lm_sensors) posebno procesora - ima dosta modernih ploca koje traze softversku kontrolu ventilatora koju vecina linux distribucija ne ukljucuje out-of-the-box (fancontrol). pregrijavanje disk kontrolera takodje ne treba iskljuciti - redovna je praksa proizvodjaca da stavljaju inferiorne hladnjake na chipset. zdravlje diskova je takodje bitno (smartd) posebno jer imas raid 1 lanac. pozeljno je da imas diskove istog proizvodjaca u lancu da ne bi "otkrio" neki sumanuti bug u kombinovanju diskova. mada, cak i u slucaju raspada lanca sistem bi "mrdao" tj. po svoj prilici vjerovatno odgovarao na ping ili neki input sa tastature. jos jedna stvar koja moze da se pregrije i proizvede taj efekat je graficka kartica. ako vec moras da vrtis X iskoristi ga da provjeris temperaturu graficke karte takodje. drugo sto mi pada na pamet je neadekvatno napajanje (bilo mala snaga PSU bilo "losa" uticnica u zidu - probaj drugu, stavi UPS) koje je uzrok pregrijavanja ili nekog drugog elektricnog problema, takodje treba obratiti i paznju na to da li sistem mozda pokusava neuspjesno da se hibernira, ode u standby, ugasi disk ili da promjeni takt procesora ili magistrale (kao powersave feature) mada to bi se vjerovatno ispoljilo vrlo brzo nakon boot-a (iskljuci svaku mogucnost za powersave u bios-u). elektricna stabilnost spoljnih komponenti je takodje bitna i posto pominjes probleme sa eth0 provjeri koliko je stabilan switch, koliko je dobar mrezni kabl, etc.

koristan softver:

lm_sensors
smartd
fancontrol

korisne bi bile sljedece informacije:

prozvodjac i model maticne ploce
prozivodjac i model napajanja, snaga
dmesg nakon boot-a
cat /proc/cpuinfo
uname -a
lspci
lsmod

Da budem detaljniji.
Server nije u mojoj firmi, već u firmi koja je kupila našu aplikaciju koja se vrti na ovim serverima. Pošto je to firma sa vrlo stogim sigurnosnim pravilima nemam pristup serverima posle radnog vremena i vikendom.
Serveri su u server room-u sa vrlo dobrim hladjenjem, nalaze se u rack-u sa još par servera na istom UPS-u.
Serveri moraju raditi u run-level-u 5 jer se još uvijek radi na modifikacijama na aplikaciji, sistem radi tek mjesec dana.
Opremu je isporučio HP, svi su diskovi isti.

Ja sam database administrator i nemam baš mnogo iskustva sa troubleshutingom problema u radu OS-a ili hardware-a, ali pošto nema neko drugi moram ja.

Pokušaću da uradim ono što mi je Aleksandar predložio. Nadam se da ću nekako doći do uzroka problema.

Što se tiče hardware-a situacija je sledeća:

Server je HP ProLiant DL380 G6 E5504, memorija HP-ova 3X2GB, HDD je isto HP-ov 6X146GB 6G SAS 10k u RAID-u 1 i napajanje je HP-ovo Power Supply 460w HE 12W Hotplug i to dva.

uh, to je totalno drugi par rukava :)
na tvom mjestu bih odmah zatrazio pomoc od distributera (on mora barem da na licu mjesta potvrdi ispravnost hardvera i verzije firmware-a diskova i raid kontrolera) i eksperimentisao sa memorijom. u biosu mozes da ukljucis "podrsku za linux" sto ce automatski prilagoditi pojedine bios opcije normalnom radu pod linuxom. u logu samog servera provjeri da li su registrovani bilo kakvi incidenti. najjednostavnije je da iz bios-a konfigurises iLO (f8 prilikom boot-a kad se ponudi iLO configuration opcija), prikljucis ga na mrezu i pregledas log (ILM) iz browsera (pozeljno IE6+). ako je server kupljen sa "Integrated Lights-Out Advanced Pack" preko iLO mozes dobiti direktan pristup virtuelnoj konzoli i u grafickom modu.
CentOS je (na moje veliko iznenadjenje) podrzan od strane HP-a tako da mozes dobiti i podrsku proizvodjaca bilo direktno ili preko distributera. mozes instalirati i HP support pack za CentOS http://www.hp.com/#Support koji je vrlo koristan ako imas probleme i koji omogucava monitoring i kontrolu hardvera kao i okruzenja (temperature). oko svega toga vjerovano mozes odmah dobiti pomoc distributera i bilo bi vrlo korisno da ga odmah uvedes u svoj problem posebno jer nisi sysadmin, a problem koji imas lici na hardver glitch.
nevezano za probleme koje imas samo bih iznio svoje misljenje glede konfiguracije servera. sa 6 diskova mogao si da napravis raid 5 od 5 diskova i sesti kao online spare, a ako je bas velika paranoja za podacima onda raid0+1 koji ima bolje performanse od raid 1 i jednaku redundantnost podataka. to sada ne mozes da promjenis bez reinstalacije os-a.
G6 je izuzetan skok u performansama u odnosu na G3 i G4 a znacajno je poboljsanje i u odnosu na G5. Sa jednim G6 sa 12GB RAM-a (verzija Proliant "Performance") imas prakticno snagu 3xG4 i prva stvar koja pada na pamet je instalacija virtuelne masine (vmware ESX) i podjela resursa. iako je ESX komercijalan softver na kraju se isplati zahvaljujuci velikim resursima servera (e.g mozes da upgrade 3 G4 na jedan G6, cijena ESX-a je oko 50% cijene servera i time si dobio 1.5 server za dzabe u startu i ne placas odrzavanje za 3 vec za jednu masinu).
u svakom slucaju ne bih iskljucio potencijalne elektricne probleme i ako su oba napajanja prikljucena na UPS, maknuo bih jedno na obicnu mrezu cisto da se izbjegne mogucnost da UPS daje losu struju (takav raspored napajanja pomaze i u slucaju crkavanja UPS-a). takodje bih iskljucio ASR i sve powersave opcije na serveru. G6 moze da kontrolise svoje PSU's i da vrlo aktivno mijenja opterecenje napajanja u zavisnosti od potreba sistema.

sretno.

E Suade, mogao si sve to lepo na pocetku da napises pa da ti Aleksandar odmah ovako lepo odgovori :)

---

Moja greška ;)

RAID1 je korisnik tražio.

Sad je kasno za virtualizaciju.

Pokušaću da nešto od ovoga uradim, jer ne zavisi sve od mene.

Trenutno radim instalaciju HP ProLiant Support Pack-a za CentOS 5. Možda mi to pomogne.

Hvala na pomoći.

Pozdrav.

nema na cemu :) ali u svakom slucaju distributera za usi ili upostavi kontakt sa hp supportom ako nastavis da imas probleme. iLO je vrlo korisna stvar inace, ukljuci ga cim se ukaze prilika za reboot servera (bar ne moras da ides do rack-a ako moras da power off/on server). ne vidim kako bi musterija mogla da primjeti razliku izmedju raid 1 i 1+0 osim eventualno u performansama sistema.

Ne mozes da imas RAID1 sa 6 diskova. Mozes RAID 0+1 ili 1+0, ali ne mozes RAID 1. Tj. mozes, ali onda imas kapacitet jednog diska na 6 mesta.

Inace, pozvati HP je sjajna ideja.

---

ne znam kako taj hp array kontroler posmatra diskove (vjerovatno nije moguce linearno povezati diskove i onda ih mirror) tj. da li je striping nezaobilazan, ali sasvim sigurno je moguce od sest diskova napraviti tri logicka drajva koji su mirrored i onda ih uvezati LVM-om ili pak ne. ne vidim zasto bi to neko radio ali vjerujem da je to jedina raid 1 kombinacija moguca na tom serveru i tako se moze dobiti kao krajnji rezultat prividno isti efekat kao da je napravljen 1+0 odnosno 0+1 od 6 diskova. performanse ovih kombinacija mogu znacajno variraju ali u ovom slucaju za sve to brine kontroler tako da je razlika neznatna ako sve povjeris kontroleru (tj. ne bavis se LVM-om) a redundantnost ista - u svakom trenutku mozes da izvuces polovinu diskova i "koprias" ih na drugi array.

Neki LSI / MegaRAID kontroleri koje sam probao su imali neki RAID 1E, tako nesto... u sustini se svodilo na RAID 10. Ti mu kazes "RAID 1" i das mu 6 diskova, a on napravi RAID 10 ili RAID 0+1 (zavisi od proizvodjaca).

---

Zdravo.

Evo opet se, nakon dvije nedelje, javio isti problem.

Sada server mogu da ping-am, kada pokušam da se konektujem pomoću PUTTY-a dozvoli mi da unesem username, a kada unesem password zaglavi i na kraju nakon 10 minuta izbaci grešku o mrežnom time-out-u.

Konfigurisao sam iLO2 ali mi ni to ništa nije pomoglo.
Evo iLO2 screen-ova za vrijme dok je server zaglavljen:



Server sam morao da resetujem ručno jer ni pomoću iLO2 konzole nisam mogao.

Evo i dijelova nekih značajnih logova:

dmesg:

ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:00:1c.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.0:pcie00]
Allocate Port Service[0000:00:1c.0:pcie03]
ACPI: PCI Interrupt 0000:00:1c.2[C] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:00:1c.2 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.2:pcie00]
Allocate Port Service[0000:00:1c.2:pcie03]
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: CPU0 (power states: C1[C1] C2[C3] C3[C3])
ACPI: CPU2 (power states: C1[C1] C2[C3] C3[C3])
ACPI: CPU1 (power states: C1[C1] C2[C3] C3[C3])
ACPI: CPU3 (power states: C1[C1] C2[C3] C3[C3])
ACPI: Thermal Zone [THM0] (8 C)
Real Time Clock Driver v1.12ac
hpet_resources: 0xfed00000 is busy
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
brd: module loaded
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Probing IDE interface ide0...
Probing IDE interface ide1...
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: PS/2 Controller [PNP0303:KBD,PNP0f0e:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
ACPI: (supports S0 S4 S5)
Initalizing network drop monitor service
Freeing unused kernel memory: 208k freed
Write protecting the kernel read-only data: 496k
GSI 17 sharing vector 0x5A and IRQ 17
ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 20 (level, low) -> IRQ 90
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 32 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 90, io mem 0xf1df0000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 20 (level, low) -> IRQ 90
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.0: irq 90, io base 0x00001000
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
GSI 18 sharing vector 0x62 and IRQ 18
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 23 (level, low) -> IRQ 98
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.1: irq 98, io base 0x00001020
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
GSI 19 sharing vector 0x6A and IRQ 19
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 22 (level, low) -> IRQ 106
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.2: irq 106, io base 0x00001040
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:1d.3[D] -> GSI 23 (level, low) -> IRQ 98
PCI: Setting latency timer of device 0000:00:1d.3 to 64
uhci_hcd 0000:00:1d.3: UHCI Host Controller
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:1d.3: irq 98, io base 0x00001060
usb usb5: configuration #1 chosen from 1 choice
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:01:04.4[B] -> GSI 22 (level, low) -> IRQ 106
uhci_hcd 0000:01:04.4: UHCI Host Controller
uhci_hcd 0000:01:04.4: new USB bus registered, assigned bus number 6
uhci_hcd 0000:01:04.4: port count misdetected? forcing to 2 ports
uhci_hcd 0000:01:04.4: irq 106, io base 0x00003800
usb usb6: configuration #1 chosen from 1 choice
hub 6-0:1.0: USB hub found
hub 6-0:1.0: 2 ports detected
SCSI subsystem initialized
HP CISS Driver (v 3.6.20-RH3)
GSI 20 sharing vector 0x72 and IRQ 20
ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 28 (level, low) -> IRQ 114
cciss0: <0x323a> at PCI 0000:04:00.0 IRQ 138 using DAC
blocks= 860051248 block_size= 512
heads= 255, sectors= 32, cylinders= 105399

blocks= 860051248 block_size= 512
heads= 255, sectors= 32, cylinders= 105399

cciss/c0d0: p1 p2
libata version 3.00 loaded.
ata_piix 0000:00:1f.2: version 2.12
GSI 21 sharing vector 0x9A and IRQ 21
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 17 (level, low) -> IRQ 154
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
usb 6-1: new full speed USB device using uhci_hcd and address 2
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: SATA max UDMA/133 cmd 0x1080 ctl 0x1088 bmdma 0x10a0 irq 154
ata2: SATA max UDMA/133 cmd 0x1090 ctl 0x1098 bmdma 0x10a8 irq 154
usb 6-1: configuration #1 chosen from 1 choice
input: HP Virtual Keyboard as /class/input/input0
input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:01:04.4-1
input: HP Virtual Keyboard as /class/input/input1
input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:01:04.4-1
usb 6-2: new full speed USB device using uhci_hcd and address 3
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.01: NODEV after polling detection
ata1.00: ATAPI: Optiarc DVD-ROM DDU820S, MH52, max UDMA/100
usb 6-2: configuration #1 chosen from 1 choice
hub 6-2:1.0: USB hub found
hub 6-2:1.0: 7 ports detected
ata1.00: configured for UDMA/100
ata2: SATA link down (SStatus 4 SControl 300)
Vendor: Optiarc Model: DVD-ROM DDU820S Rev: MH52
Type: CD-ROM ANSI SCSI revision: 05
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: [email protected]
device-mapper: dm-raid45: initialized v0.2594l
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: dm-0: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 91095045
ext3_orphan_cleanup: deleting unreferenced inode 91095046
ext3_orphan_cleanup: deleting unreferenced inode 91095047
ext3_orphan_cleanup: deleting unreferenced inode 91095056
ext3_orphan_cleanup: deleting unreferenced inode 91095059
ext3_orphan_cleanup: deleting unreferenced inode 91095064
EXT3-fs: dm-0: 6 orphan inodes deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: Disabled at runtime.
SELinux: Unregistering netfilter hooks
type=1404 audit(1265368482.149:2): selinux=0 auid=4294967295 ses=4294967295
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.9.3 (March 17, 2009)
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:02:00.0 to 64
eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f4000000, IRQ 169, node addr 0025b3deeacc
ACPI: PCI Interrupt 0000:02:00.1[B] -> GSI 17 (level, low) -> IRQ 154
PCI: Setting latency timer of device 0000:02:00.1 to 64
eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f2000000, IRQ 154, node addr 0025b3deeace
GSI 22 sharing vector 0xA2 and IRQ 22
ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 18 (level, low) -> IRQ 162
PCI: Setting latency timer of device 0000:03:00.0 to 64
eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 162, node addr 0025b3deead0
GSI 23 sharing vector 0xAA and IRQ 23
ACPI: PCI Interrupt 0000:03:00.1[B] -> GSI 19 (level, low) -> IRQ 170
PCI: Setting latency timer of device 0000:03:00.1 to 64
eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f6000000, IRQ 170, node addr 0025b3deead2
Floppy drive(s): fd0 is 1.44M
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
ACPI: PCI Interrupt 0000:01:04.2[B] -> GSI 22 (level, low) -> IRQ 106
input: PC Speaker as /class/input/input2
scsi 0:0:0:0: Attached scsi generic sg0 type 5
sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 0:0:0:0: Attached scsi CD-ROM sr0
floppy0: no floppy controllers found
Floppy drive(s): fd0 is 1.44M
floppy0: no floppy controllers found
lp: driver loaded but no devices found
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
mtrr: type mismatch for e8000000,4000000 old: uncachable new: write-combining
ACPI: Power Button (FF) [PWRF]
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
device-mapper: multipath: version 1.0.5 loaded
EXT3 FS on dm-0, internal journal
kjournald starting. Commit interval 5 seconds
EXT3 FS on cciss/c0d0p1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 8159224k swap on /dev/VolGroup00/LogVol01. Priority:-1 extents:1 across:8159224k
IA-32 Microcode Update Driver: v1.14a <[email protected]>
bnx2: eth0: using MSIX
ADDRCONF(NETDEV_UP): eth0: link is not ready
bnx2: eth0 NIC Copper Link is Up, 100 Mbps full duplex
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
bnx2: eth1: using MSIX
ADDRCONF(NETDEV_UP): eth1: link is not ready
bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
eth0: no IPv6 routers present
Installing knfsd (copyright (C) 1996 [email protected]).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
eth1: no IPv6 routers present
mtrr: type mismatch for e8000000,4000000 old: uncachable new: write-combining


messages:


Feb 2 12:18:21 ucmserver NXSERVER-3.4.0-8[21312]: Session with id: A9A3E388E357A5F96D95FB490CAE9693: restored 'NXShell::handler_session_start'
Feb 2 12:18:22 ucmserver gconfd (oracle-21318): starting (version 2.14.0), pid 21318 user 'oracle'
Feb 2 12:18:22 ucmserver gconfd (oracle-21318): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
Feb 2 12:18:22 ucmserver gconfd (oracle-21318): Resolved address "xml:readwrite:/home/oracle/.gconf" to a writable configuration source at position 1
Feb 2 12:18:22 ucmserver gconfd (oracle-21318): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
Feb 5 12:15:09 ucmserver syslogd 1.4.1: restart.
Feb 5 12:15:09 ucmserver kernel: klogd 1.4.1, log source = /proc/kmsg started.
Feb 5 12:15:09 ucmserver kernel: Linux version 2.6.18-164.el5 ([email protected]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Thu Sep 3 03:28:30 EDT 2009
Feb 5 12:15:09 ucmserver kernel: Command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet
Feb 5 12:15:09 ucmserver kernel: BIOS-provided physical RAM map:
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 0000000000010000 - 000000000009f400 (usable)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 0000000000100000 - 00000000df63f000 (usable)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000df63f000 - 00000000df64c000 (ACPI data)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000df64c000 - 00000000df64d000 (usable)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000df64d000 - 00000000e4000000 (reserved)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000fec00000 - 00000000fee10000 (reserved)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 0000000100000000 - 000000019ffff000 (usable)
Feb 5 12:15:09 ucmserver kernel: DMI 2.6 present.
Feb 5 12:15:09 ucmserver rpc.statd[2735]: Version 1.0.9 Starting
Feb 5 12:15:09 ucmserver kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Feb 5 12:15:09 ucmserver kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Feb 5 12:15:09 ucmserver kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Feb 5 12:15:09 ucmserver kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Feb 5 12:15:09 ucmserver kernel: SRAT: Node 0 PXM 0 0-e0000000
Feb 5 12:15:09 ucmserver kernel: SRAT: Node 0 PXM 0 0-1a0000000
Feb 5 12:15:09 ucmserver kernel: Bootmem setup node 0 0000000000000000-000000019ffff000
Feb 5 12:15:09 ucmserver kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Feb 5 12:15:09 ucmserver kernel: disabling kdump
Feb 5 12:15:10 ucmserver rpc.statd[2735]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user
Feb 5 12:15:10 ucmserver kernel: ACPI: PM-Timer IO Port: 0x908
Feb 5 12:15:10 ucmserver kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Feb 5 12:15:10 ucmserver kernel: Processor #0 7:10 APIC version 21
...
Feb 5 12:15:10 ucmserver kernel: Setting APIC routing to physical flat
Feb 5 12:15:10 ucmserver kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000
Feb 5 12:15:10 ucmserver kernel: Using ACPI (MADT) for SMP configuration information
Feb 5 12:15:10 ucmserver kernel: Nosave address range: 000000000009f000 - 00000000000a0000
Feb 5 12:15:10 ucmserver pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.
Feb 5 12:15:10 ucmserver kernel: Nosave address range: 00000000000a0000 - 00000000000f0000
Feb 5 12:15:11 ucmserver pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)
...
Feb 5 12:15:11 ucmserver kernel: Security Framework v1.0.0 initialized
Feb 5 12:15:11 ucmserver kernel: SELinux: Initializing.
Feb 5 12:15:11 ucmserver kernel: selinux_register_security: Registering secondary module capability
Feb 5 12:15:11 ucmserver kernel: Capability LSM initialized as secondary
Feb 5 12:15:11 ucmserver kernel: Mount-cache hash table entries: 256
Feb 5 12:15:11 ucmserver kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Feb 5 12:15:11 ucmserver kernel: CPU: L2 cache: 256K
Feb 5 12:15:11 ucmserver kernel: CPU: L3 cache: 4096K
Feb 5 12:15:11 ucmserver kernel: CPU 0/0 -> Node 0
Feb 5 12:15:11 ucmserver kernel: using mwait in idle threads.
Feb 5 12:15:11 ucmserver kernel: CPU: Physical Processor ID: 0
Feb 5 12:15:11 ucmserver kernel: CPU: Processor Core ID: 0
Feb 5 12:15:11 ucmserver kernel: CPU0: Thermal monitoring enabled (TM1)
Feb 5 12:15:11 ucmserver kernel: SMP alternatives: switching to UP code
Feb 5 12:15:11 ucmserver kernel: ACPI: Core revision 20060707
Feb 5 12:15:11 ucmserver kernel: Using local APIC timer interrupts.
Feb 5 12:15:11 ucmserver kernel: result 8333813
Feb 5 12:15:11 ucmserver kernel: Detected 8.333 MHz APIC timer.
Feb 5 12:15:11 ucmserver kernel: SMP alternatives: switching to SMP code
Feb 5 12:15:12 ucmserver kernel: Booting processor 1/4 APIC 0x4
Feb 5 12:15:12 ucmserver kernel: Initializing CPU#1
Feb 5 12:15:12 ucmserver kernel: Calibrating delay using timer specific routine.. 4000.05 BogoMIPS (lpj=2000028)
Feb 5 12:15:12 ucmserver kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Feb 5 12:15:12 ucmserver kernel: CPU: L2 cache: 256K
Feb 5 12:15:12 ucmserver kernel: CPU: L3 cache: 4096K
Feb 5 12:15:12 ucmserver kernel: CPU 1/4 -> Node 0
Feb 5 12:15:12 ucmserver kernel: CPU: Physical Processor ID: 0
Feb 5 12:15:12 ucmserver kernel: CPU: Processor Core ID: 2
Feb 5 12:15:12 ucmserver kernel: CPU1: Thermal monitoring enabled (TM1)
Feb 5 12:15:12 ucmserver kernel: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz stepping 05
Feb 5 12:15:12 ucmserver kernel: SMP alternatives: switching to SMP code
Feb 5 12:15:12 ucmserver kernel: Booting processor 2/4 APIC 0x2
Feb 5 12:15:12 ucmserver kernel: Initializing CPU#2
Feb 5 12:15:12 ucmserver kernel: Calibrating delay using timer specific routine.. 4000.23 BogoMIPS (lpj=2000117)
Feb 5 12:15:12 ucmserver kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Feb 5 12:15:12 ucmserver kernel: CPU: L2 cache: 256K
Feb 5 12:15:12 ucmserver kernel: CPU: L3 cache: 4096K
Feb 5 12:15:12 ucmserver kernel: CPU 2/2 -> Node 0
...
Feb 5 12:15:13 ucmserver kernel: DMAR:RMRR base: 0x00000000df7fc000 end: 0x00000000df7fdfff
Feb 5 12:15:13 ucmserver kernel: DMAR:RMRR base: 0x00000000df7f5000 end: 0x00000000df7fafff
Feb 5 12:15:13 ucmserver kernel: DMAR:RMRR base: 0x00000000df64e000 end: 0x00000000df64ffff
Feb 5 12:15:13 ucmserver kernel: DMAR:Unknown DMAR structure type
Feb 5 12:15:13 ucmserver kernel: PCI-GART: No AMD northbridge found.
Feb 5 12:15:13 ucmserver kernel: pnp: 00:01: ioport range 0x408-0x40f has been reserved
Feb 5 12:15:13 ucmserver kernel: pnp: 00:01: ioport range 0x4d0-0x4d1 has been reserved
Feb 5 12:15:13 ucmserver kernel: pnp: 00:01: ioport range 0x700-0x71f has been reserved
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:01.0
Feb 5 12:15:13 ucmserver kernel: IO window: 4000-4fff
Feb 5 12:15:13 ucmserver kernel: MEM window: fbb00000-fbffffff
Feb 5 12:15:13 ucmserver kernel: PREFETCH window 0x00000000e6000000-0x00000000e60fffff
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:02.0
...
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:02.0
Feb 5 12:15:13 ucmserver kernel: IO window: disabled.
Feb 5 12:15:13 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:13 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:03.0
Feb 5 12:15:13 ucmserver kernel: IO window: disabled.
Feb 5 12:15:13 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:13 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:04.0
Feb 5 12:15:13 ucmserver kernel: IO window: disabled.
Feb 5 12:15:13 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:13 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:05.0
Feb 5 12:15:13 ucmserver kernel: IO window: disabled.
Feb 5 12:15:13 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:14 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:14 ucmserver kernel: PCI: Bridge: 0000:00:06.0
Feb 5 12:15:14 ucmserver kernel: IO window: disabled.
Feb 5 12:15:14 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:14 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:14 ucmserver kernel: PCI: Bridge: 0000:00:07.0
...
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).
Feb 5 12:15:20 ucmserver kernel: EXT3-fs: write access will be enabled during recovery.
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: Successfully dropped root privileges.
Feb 5 12:15:20 ucmserver kernel: kjournald starting. Commit interval 5 seconds
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: avahi-daemon 0.6.16 starting up.
Feb 5 12:15:20 ucmserver kernel: EXT3-fs: dm-0: orphan cleanup on readonly fs
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: Successfully called chroot().
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: Successfully dropped remaining capabilities.
...
Feb 5 12:15:39 ucmserver smartd[3485]: smartd version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Feb 5 12:15:39 ucmserver smartd[3485]: Home page is http://smartmontools.sourceforge.net/
Feb 5 12:15:39 ucmserver smartd[3485]: Opened configuration file /etc/smartd.conf
Feb 5 12:15:39 ucmserver smartd[3485]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Feb 5 12:15:39 ucmserver smartd[3485]: Problem creating device name scan list
Feb 5 12:15:39 ucmserver smartd[3485]: Problem creating device name scan list
Feb 5 12:15:39 ucmserver smartd[3485]: In the system's table of devices NO devices found to scan
Feb 5 12:15:39 ucmserver smartd[3485]: Monitoring 0 ATA and 0 SCSI devices
Feb 5 12:15:39 ucmserver smartd[3487]: smartd has fork()ed into background mode. New PID=3487.
Feb 5 12:15:40 ucmserver pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
Feb 5 12:15:40 ucmserver last message repeated 3 times
Feb 5 12:15:42 ucmserver kernel: mtrr: type mismatch for e8000000,4000000 old: uncachable new: write-combining




Zašto OS vidi samo 5GB RAM-a?

free -g
total used free shared buffers cached
Mem: 5 0 4 0 0 0
-/+ buffers/cache: 0 5
Swap: 7 0 7

Kad je:

cat /proc/meminfo |grep -i memtotal
MemTotal: 6103680 kB

Zdravo.

Evo opet se, nakon dvije nedelje, javio isti problem.

Sada server mogu da ping-am, kada pokušam da se konektujem pomoću PUTTY-a dozvoli mi da unesem username, a kada unesem password zaglavi i na kraju nakon 10 minuta izbaci grešku o mrežnom time-out-u.

Konfigurisao sam iLO2 ali mi ni to ništa nije pomoglo.
Evo iLO2 screen-ova za vrijme dok je server zaglavljen:

















Server sam morao da resetujem ručno jer ni pomoću iLO2 konzole nisam mogao.

Evo i dijelova nekih značajnih logova:

dmesg:

ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:00:1c.0 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.0:pcie00]
Allocate Port Service[0000:00:1c.0:pcie03]
ACPI: PCI Interrupt 0000:00:1c.2[C] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:00:1c.2 to 64
assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:1c.2:pcie00]
Allocate Port Service[0000:00:1c.2:pcie03]
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: CPU0 (power states: C1[C1] C2[C3] C3[C3])
ACPI: CPU2 (power states: C1[C1] C2[C3] C3[C3])
ACPI: CPU1 (power states: C1[C1] C2[C3] C3[C3])
ACPI: CPU3 (power states: C1[C1] C2[C3] C3[C3])
ACPI: Thermal Zone [THM0] (8 C)
Real Time Clock Driver v1.12ac
hpet_resources: 0xfed00000 is busy
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:09: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
brd: module loaded
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Probing IDE interface ide0...
Probing IDE interface ide1...
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: PS/2 Controller [PNP0303:KBD,PNP0f0e:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
ACPI: (supports S0 S4 S5)
Initalizing network drop monitor service
Freeing unused kernel memory: 208k freed
Write protecting the kernel read-only data: 496k
GSI 17 sharing vector 0x5A and IRQ 17
ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 20 (level, low) -> IRQ 90
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1d.7: debug port 1
PCI: cache line size of 32 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: irq 90, io mem 0xf1df0000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
ohci_hcd: 2005 April 22 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
USB Universal Host Controller Interface driver v3.0
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 20 (level, low) -> IRQ 90
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.0: irq 90, io base 0x00001000
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
GSI 18 sharing vector 0x62 and IRQ 18
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 23 (level, low) -> IRQ 98
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.1: irq 98, io base 0x00001020
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
GSI 19 sharing vector 0x6A and IRQ 19
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 22 (level, low) -> IRQ 106
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.2: irq 106, io base 0x00001040
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:1d.3[D] -> GSI 23 (level, low) -> IRQ 98
PCI: Setting latency timer of device 0000:00:1d.3 to 64
uhci_hcd 0000:00:1d.3: UHCI Host Controller
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:1d.3: irq 98, io base 0x00001060
usb usb5: configuration #1 chosen from 1 choice
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:01:04.4[B] -> GSI 22 (level, low) -> IRQ 106
uhci_hcd 0000:01:04.4: UHCI Host Controller
uhci_hcd 0000:01:04.4: new USB bus registered, assigned bus number 6
uhci_hcd 0000:01:04.4: port count misdetected? forcing to 2 ports
uhci_hcd 0000:01:04.4: irq 106, io base 0x00003800
usb usb6: configuration #1 chosen from 1 choice
hub 6-0:1.0: USB hub found
hub 6-0:1.0: 2 ports detected
SCSI subsystem initialized
HP CISS Driver (v 3.6.20-RH3)
GSI 20 sharing vector 0x72 and IRQ 20
ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 28 (level, low) -> IRQ 114
cciss0: <0x323a> at PCI 0000:04:00.0 IRQ 138 using DAC
blocks= 860051248 block_size= 512
heads= 255, sectors= 32, cylinders= 105399

blocks= 860051248 block_size= 512
heads= 255, sectors= 32, cylinders= 105399

cciss/c0d0: p1 p2
libata version 3.00 loaded.
ata_piix 0000:00:1f.2: version 2.12
GSI 21 sharing vector 0x9A and IRQ 21
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 17 (level, low) -> IRQ 154
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
usb 6-1: new full speed USB device using uhci_hcd and address 2
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi0 : ata_piix
scsi1 : ata_piix
ata1: SATA max UDMA/133 cmd 0x1080 ctl 0x1088 bmdma 0x10a0 irq 154
ata2: SATA max UDMA/133 cmd 0x1090 ctl 0x1098 bmdma 0x10a8 irq 154
usb 6-1: configuration #1 chosen from 1 choice
input: HP Virtual Keyboard as /class/input/input0
input: USB HID v1.01 Keyboard [HP Virtual Keyboard] on usb-0000:01:04.4-1
input: HP Virtual Keyboard as /class/input/input1
input: USB HID v1.01 Mouse [HP Virtual Keyboard] on usb-0000:01:04.4-1
usb 6-2: new full speed USB device using uhci_hcd and address 3
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.01: NODEV after polling detection
ata1.00: ATAPI: Optiarc DVD-ROM DDU820S, MH52, max UDMA/100
usb 6-2: configuration #1 chosen from 1 choice
hub 6-2:1.0: USB hub found
hub 6-2:1.0: 7 ports detected
ata1.00: configured for UDMA/100
ata2: SATA link down (SStatus 4 SControl 300)
Vendor: Optiarc Model: DVD-ROM DDU820S Rev: MH52
Type: CD-ROM ANSI SCSI revision: 05
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: [email protected]
device-mapper: dm-raid45: initialized v0.2594l
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: dm-0: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 91095045
ext3_orphan_cleanup: deleting unreferenced inode 91095046
ext3_orphan_cleanup: deleting unreferenced inode 91095047
ext3_orphan_cleanup: deleting unreferenced inode 91095056
ext3_orphan_cleanup: deleting unreferenced inode 91095059
ext3_orphan_cleanup: deleting unreferenced inode 91095064
EXT3-fs: dm-0: 6 orphan inodes deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux: Disabled at runtime.
SELinux: Unregistering netfilter hooks
type=1404 audit(1265368482.149:2): selinux=0 auid=4294967295 ses=4294967295
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.9.3 (March 17, 2009)
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:02:00.0 to 64
eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f4000000, IRQ 169, node addr 0025b3deeacc
ACPI: PCI Interrupt 0000:02:00.1[B] -> GSI 17 (level, low) -> IRQ 154
PCI: Setting latency timer of device 0000:02:00.1 to 64
eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f2000000, IRQ 154, node addr 0025b3deeace
GSI 22 sharing vector 0xA2 and IRQ 22
ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 18 (level, low) -> IRQ 162
PCI: Setting latency timer of device 0000:03:00.0 to 64
eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f8000000, IRQ 162, node addr 0025b3deead0
GSI 23 sharing vector 0xAA and IRQ 23
ACPI: PCI Interrupt 0000:03:00.1[B] -> GSI 19 (level, low) -> IRQ 170
PCI: Setting latency timer of device 0000:03:00.1 to 64
eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f6000000, IRQ 170, node addr 0025b3deead2
Floppy drive(s): fd0 is 1.44M
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
ACPI: PCI Interrupt 0000:01:04.2[B] -> GSI 22 (level, low) -> IRQ 106
input: PC Speaker as /class/input/input2
scsi 0:0:0:0: Attached scsi generic sg0 type 5
sr0: scsi3-mmc drive: 24x/24x cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.20
sr 0:0:0:0: Attached scsi CD-ROM sr0
floppy0: no floppy controllers found
Floppy drive(s): fd0 is 1.44M
floppy0: no floppy controllers found
lp: driver loaded but no devices found
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
IPv6 over IPv4 tunneling driver
mtrr: type mismatch for e8000000,4000000 old: uncachable new: write-combining
ACPI: Power Button (FF) [PWRF]
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
device-mapper: multipath: version 1.0.5 loaded
EXT3 FS on dm-0, internal journal
kjournald starting. Commit interval 5 seconds
EXT3 FS on cciss/c0d0p1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
Adding 8159224k swap on /dev/VolGroup00/LogVol01. Priority:-1 extents:1 across:8159224k
IA-32 Microcode Update Driver: v1.14a <[email protected]>
bnx2: eth0: using MSIX
ADDRCONF(NETDEV_UP): eth0: link is not ready
bnx2: eth0 NIC Copper Link is Up, 100 Mbps full duplex
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
bnx2: eth1: using MSIX
ADDRCONF(NETDEV_UP): eth1: link is not ready
bnx2: eth1 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
eth0: no IPv6 routers present
Installing knfsd (copyright (C) 1996 [email protected]).
NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
NFSD: starting 90-second grace period
eth1: no IPv6 routers present
mtrr: type mismatch for e8000000,4000000 old: uncachable new: write-combining


messages:


Feb 2 12:18:21 ucmserver NXSERVER-3.4.0-8[21312]: Session with id: A9A3E388E357A5F96D95FB490CAE9693: restored 'NXShell::handler_session_start'
Feb 2 12:18:22 ucmserver gconfd (oracle-21318): starting (version 2.14.0), pid 21318 user 'oracle'
Feb 2 12:18:22 ucmserver gconfd (oracle-21318): Resolved address "xml:readonly:/etc/gconf/gconf.xml.mandatory" to a read-only configuration source at position 0
Feb 2 12:18:22 ucmserver gconfd (oracle-21318): Resolved address "xml:readwrite:/home/oracle/.gconf" to a writable configuration source at position 1
Feb 2 12:18:22 ucmserver gconfd (oracle-21318): Resolved address "xml:readonly:/etc/gconf/gconf.xml.defaults" to a read-only configuration source at position 2
Feb 5 12:15:09 ucmserver syslogd 1.4.1: restart.
Feb 5 12:15:09 ucmserver kernel: klogd 1.4.1, log source = /proc/kmsg started.
Feb 5 12:15:09 ucmserver kernel: Linux version 2.6.18-164.el5 ([email protected]) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Thu Sep 3 03:28:30 EDT 2009
Feb 5 12:15:09 ucmserver kernel: Command line: ro root=/dev/VolGroup00/LogVol00 rhgb quiet
Feb 5 12:15:09 ucmserver kernel: BIOS-provided physical RAM map:
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 0000000000010000 - 000000000009f400 (usable)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 0000000000100000 - 00000000df63f000 (usable)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000df63f000 - 00000000df64c000 (ACPI data)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000df64c000 - 00000000df64d000 (usable)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000df64d000 - 00000000e4000000 (reserved)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 00000000fec00000 - 00000000fee10000 (reserved)
Feb 5 12:15:09 ucmserver kernel: BIOS-e820: 0000000100000000 - 000000019ffff000 (usable)
Feb 5 12:15:09 ucmserver kernel: DMI 2.6 present.
Feb 5 12:15:09 ucmserver rpc.statd[2735]: Version 1.0.9 Starting
Feb 5 12:15:09 ucmserver kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Feb 5 12:15:09 ucmserver kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Feb 5 12:15:09 ucmserver kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Feb 5 12:15:09 ucmserver kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Feb 5 12:15:09 ucmserver kernel: SRAT: Node 0 PXM 0 0-e0000000
Feb 5 12:15:09 ucmserver kernel: SRAT: Node 0 PXM 0 0-1a0000000
Feb 5 12:15:09 ucmserver kernel: Bootmem setup node 0 0000000000000000-000000019ffff000
Feb 5 12:15:09 ucmserver kernel: Memory for crash kernel (0x0 to 0x0) notwithin permissible range
Feb 5 12:15:09 ucmserver kernel: disabling kdump
Feb 5 12:15:10 ucmserver rpc.statd[2735]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user
Feb 5 12:15:10 ucmserver kernel: ACPI: PM-Timer IO Port: 0x908
Feb 5 12:15:10 ucmserver kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Feb 5 12:15:10 ucmserver kernel: Processor #0 7:10 APIC version 21
...
Feb 5 12:15:10 ucmserver kernel: Setting APIC routing to physical flat
Feb 5 12:15:10 ucmserver kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000
Feb 5 12:15:10 ucmserver kernel: Using ACPI (MADT) for SMP configuration information
Feb 5 12:15:10 ucmserver kernel: Nosave address range: 000000000009f000 - 00000000000a0000
Feb 5 12:15:10 ucmserver pcscd: hotplug_libusb.c:402:HPEstablishUSBNotifications() Driver ifd-egate.bundle does not support IFD_GENERATE_HOTPLUG. Using active polling instead.
Feb 5 12:15:10 ucmserver kernel: Nosave address range: 00000000000a0000 - 00000000000f0000
Feb 5 12:15:11 ucmserver pcscd: hotplug_libusb.c:411:HPEstablishUSBNotifications() Polling forced every 1 second(s)
...
Feb 5 12:15:11 ucmserver kernel: Security Framework v1.0.0 initialized
Feb 5 12:15:11 ucmserver kernel: SELinux: Initializing.
Feb 5 12:15:11 ucmserver kernel: selinux_register_security: Registering secondary module capability
Feb 5 12:15:11 ucmserver kernel: Capability LSM initialized as secondary
Feb 5 12:15:11 ucmserver kernel: Mount-cache hash table entries: 256
Feb 5 12:15:11 ucmserver kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Feb 5 12:15:11 ucmserver kernel: CPU: L2 cache: 256K
Feb 5 12:15:11 ucmserver kernel: CPU: L3 cache: 4096K
Feb 5 12:15:11 ucmserver kernel: CPU 0/0 -> Node 0
Feb 5 12:15:11 ucmserver kernel: using mwait in idle threads.
Feb 5 12:15:11 ucmserver kernel: CPU: Physical Processor ID: 0
Feb 5 12:15:11 ucmserver kernel: CPU: Processor Core ID: 0
Feb 5 12:15:11 ucmserver kernel: CPU0: Thermal monitoring enabled (TM1)
Feb 5 12:15:11 ucmserver kernel: SMP alternatives: switching to UP code
Feb 5 12:15:11 ucmserver kernel: ACPI: Core revision 20060707
Feb 5 12:15:11 ucmserver kernel: Using local APIC timer interrupts.
Feb 5 12:15:11 ucmserver kernel: result 8333813
Feb 5 12:15:11 ucmserver kernel: Detected 8.333 MHz APIC timer.
Feb 5 12:15:11 ucmserver kernel: SMP alternatives: switching to SMP code
Feb 5 12:15:12 ucmserver kernel: Booting processor 1/4 APIC 0x4
Feb 5 12:15:12 ucmserver kernel: Initializing CPU#1
Feb 5 12:15:12 ucmserver kernel: Calibrating delay using timer specific routine.. 4000.05 BogoMIPS (lpj=2000028)
Feb 5 12:15:12 ucmserver kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Feb 5 12:15:12 ucmserver kernel: CPU: L2 cache: 256K
Feb 5 12:15:12 ucmserver kernel: CPU: L3 cache: 4096K
Feb 5 12:15:12 ucmserver kernel: CPU 1/4 -> Node 0
Feb 5 12:15:12 ucmserver kernel: CPU: Physical Processor ID: 0
Feb 5 12:15:12 ucmserver kernel: CPU: Processor Core ID: 2
Feb 5 12:15:12 ucmserver kernel: CPU1: Thermal monitoring enabled (TM1)
Feb 5 12:15:12 ucmserver kernel: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz stepping 05
Feb 5 12:15:12 ucmserver kernel: SMP alternatives: switching to SMP code
Feb 5 12:15:12 ucmserver kernel: Booting processor 2/4 APIC 0x2
Feb 5 12:15:12 ucmserver kernel: Initializing CPU#2
Feb 5 12:15:12 ucmserver kernel: Calibrating delay using timer specific routine.. 4000.23 BogoMIPS (lpj=2000117)
Feb 5 12:15:12 ucmserver kernel: CPU: L1 I cache: 32K, L1 D cache: 32K
Feb 5 12:15:12 ucmserver kernel: CPU: L2 cache: 256K
Feb 5 12:15:12 ucmserver kernel: CPU: L3 cache: 4096K
Feb 5 12:15:12 ucmserver kernel: CPU 2/2 -> Node 0
...
Feb 5 12:15:13 ucmserver kernel: DMAR:RMRR base: 0x00000000df7fc000 end: 0x00000000df7fdfff
Feb 5 12:15:13 ucmserver kernel: DMAR:RMRR base: 0x00000000df7f5000 end: 0x00000000df7fafff
Feb 5 12:15:13 ucmserver kernel: DMAR:RMRR base: 0x00000000df64e000 end: 0x00000000df64ffff
Feb 5 12:15:13 ucmserver kernel: DMAR:Unknown DMAR structure type
Feb 5 12:15:13 ucmserver kernel: PCI-GART: No AMD northbridge found.
Feb 5 12:15:13 ucmserver kernel: pnp: 00:01: ioport range 0x408-0x40f has been reserved
Feb 5 12:15:13 ucmserver kernel: pnp: 00:01: ioport range 0x4d0-0x4d1 has been reserved
Feb 5 12:15:13 ucmserver kernel: pnp: 00:01: ioport range 0x700-0x71f has been reserved
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:01.0
Feb 5 12:15:13 ucmserver kernel: IO window: 4000-4fff
Feb 5 12:15:13 ucmserver kernel: MEM window: fbb00000-fbffffff
Feb 5 12:15:13 ucmserver kernel: PREFETCH window 0x00000000e6000000-0x00000000e60fffff
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:02.0
...
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:02.0
Feb 5 12:15:13 ucmserver kernel: IO window: disabled.
Feb 5 12:15:13 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:13 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:03.0
Feb 5 12:15:13 ucmserver kernel: IO window: disabled.
Feb 5 12:15:13 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:13 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:04.0
Feb 5 12:15:13 ucmserver kernel: IO window: disabled.
Feb 5 12:15:13 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:13 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:13 ucmserver kernel: PCI: Bridge: 0000:00:05.0
Feb 5 12:15:13 ucmserver kernel: IO window: disabled.
Feb 5 12:15:13 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:14 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:14 ucmserver kernel: PCI: Bridge: 0000:00:06.0
Feb 5 12:15:14 ucmserver kernel: IO window: disabled.
Feb 5 12:15:14 ucmserver kernel: MEM window: disabled.
Feb 5 12:15:14 ucmserver kernel: PREFETCH window: disabled.
Feb 5 12:15:14 ucmserver kernel: PCI: Bridge: 0000:00:07.0
...
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70).
Feb 5 12:15:20 ucmserver kernel: EXT3-fs: write access will be enabled during recovery.
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: Successfully dropped root privileges.
Feb 5 12:15:20 ucmserver kernel: kjournald starting. Commit interval 5 seconds
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: avahi-daemon 0.6.16 starting up.
Feb 5 12:15:20 ucmserver kernel: EXT3-fs: dm-0: orphan cleanup on readonly fs
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: Successfully called chroot().
Feb 5 12:15:20 ucmserver avahi-daemon[3154]: Successfully dropped remaining capabilities.
...
Feb 5 12:15:39 ucmserver smartd[3485]: smartd version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Feb 5 12:15:39 ucmserver smartd[3485]: Home page is http://smartmontools.sourceforge.net/
Feb 5 12:15:39 ucmserver smartd[3485]: Opened configuration file /etc/smartd.conf
Feb 5 12:15:39 ucmserver smartd[3485]: Configuration file /etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Feb 5 12:15:39 ucmserver smartd[3485]: Problem creating device name scan list
Feb 5 12:15:39 ucmserver smartd[3485]: Problem creating device name scan list
Feb 5 12:15:39 ucmserver smartd[3485]: In the system's table of devices NO devices found to scan
Feb 5 12:15:39 ucmserver smartd[3485]: Monitoring 0 ATA and 0 SCSI devices
Feb 5 12:15:39 ucmserver smartd[3487]: smartd has fork()ed into background mode. New PID=3487.
Feb 5 12:15:40 ucmserver pcscd: winscard.c:304:SCardConnect() Reader E-Gate 0 0 Not Found
Feb 5 12:15:40 ucmserver last message repeated 3 times
Feb 5 12:15:42 ucmserver kernel: mtrr: type mismatch for e8000000,4000000 old: uncachable new: write-combining




Zašto OS vidi samo 5GB RAM-a?

free -g
total used free shared buffers cached
Mem: 5 0 4 0 0 0
-/+ buffers/cache: 0 5
Swap: 7 0 7

Kad je:

cat /proc/meminfo |grep -i memtotal
MemTotal: 6103680 kB




_{[Ovu poruku je menjao suadhm dana 05.02.2010. u 16:04 GMT+1]}

Ustanovio sam da se problem javlja svake dvije nedelje.

Ima li neko ideju šta mogu da uradim?

Ako si otklonio sumnju na hardware, onda deinstaliraj vasu aplikaciju. ;-)

Salu na stranu, ali imao sam slicnih problema, doduse na HP-UX na Itaniumu, gde aplikacija vremenom "pojede" sve resurse i onda ti ne ostaje nista drugo nego hardware reset.

Ne znam sta tvoja aplikacija konkretno radi, ali bi ti savetovao da uzmes/napises neki monitoring script koji ce ti svaki sat vremena ocitati razne systemske resurce (memorija, razni handle-ovi, descriptori, file system,...) i da ih potom zapise u file. Onda ces moci da pratis koji resurs ti ponestaje.

Javi sta si uradio.

Situacija je sledeća:

Postoje dva servera, jedan sa Oracle bazom i drugi sa Oracle UCM software-om (Oracle-ov Software za document managment).

Server sa UCM-om (aplikativni server) zaglavljuje duplo češće od servera sa bazom.

Trenutno niko ne radi sa UCM aplikaciom, jer imaju neki problem sa mrežom, tj. nemaju kompletniranu mrežu, i na tome se radi.

Sa serverom na kojem je baza korisnici (dva korisnika) rade testno sa drugom aplikaciom koja je radjena u .Net-u i koja je instalirana na korisničkim radnim stanicama.

Znači UCM server skoro i da ništa ne radi, samo kolega koji radi na custom-izaciji UCM-a ponekad pokrene aplikaciju. Od njega i dobijam informacije kada je server zaglavio.

Jedino što sam na serverima instalirao osim OS-a, UCM-a, odnosno Oracle Baze je i NX software za remote desktop pristup na servere.


U svojoj firmi imam (razvojni) server sa istim OS-om, istom bazom i istim UCM-om, samo je 32-bitna verzija i to radi bez problema već 4 mjeseca. Takodje, i u firmi intezvino (sa velikim brojem dokumenata) koristimo UCM (interno) već 2 mjeseca bez ikakvih problema (OS je Windows 2003 Server, ista verzija baze i UCM-a).



Jedino što sada sumnjam je da postoji neki problem sa UPS-om na kojem su serveri. Kada god pridjem serverima, radi restarta, osetim blag strujni udar.


_{[Ovu poruku je menjao suadhm dana 19.02.2010. u 15:26 GMT+1]}

Problem se ponovio samo nakon dva dana.
Očigledno ne postoji nikakav obrazac u ponavljanju problema.

Danas ću probati da premostim napajanje na drugi UPS ili direktno na utičnicu.

Zbog ovakvih problema se i kupuje RHEL. Onda lepo kontaktiras RH podrsku i oni ti dodele inzenjera koji zajedno sa tobom radi na resavanju tog problema.
CentOS je dobar za testing i kucnu upotrebu.

To si upravu. Ali korisnik je zahtijevao samo CentOS, jer imaju još par servera sa tim OS-om.
Nisam siguran da je OS problem.
Pošto je isti problem na oba servera najverovatnije je do napajanja. Servere sam skinuo sa UPS-a i sad čekam da vidim da li će se problem ponoviti.

U postu od prije dva dana sam rekao da se problem opet ponovio ali je na kraju je ispalo da je neki problem sa mrežom.