Oracle Application 11i Cloning

 Version Details :
=============

OS Version      : OEL 5 Update 4
APPS Version :  11.5.10

 1. Adpreclone Step in Source APPS
========================

[applcln@test cln_test]$ pwd
/test/applmgr/uatcomn/admin/scripts/cln_test

[applcln@test cln_test]$ perl adpreclone.pl appsTier
Running Rapid Clone with command...
        perl /test/applmgr/uatappl/ad/11.5.0/bin/adclone.pl java=/test/applmgr/uatora/iAS/appsoui/jre/1.3.1 mode=stage stage=/test/applmgr/uatcomn/clone component=appsTier method=CUSTOM appctx=/test/applmgr/uatappl/admin/cln_test.xml  showProgress

Beginning application tier Stage - Fri Jan  3 16:41:13 2014
Log file located at /test/applmgr/uatappl/admin/cln_test/log/StageAppsTier_01030441.log
 
Completed Stage...
Fri Jan  3 16:43:24 2014

[applcln@test cln_test]$

2. Check Pre-requirement for 9i Support in Target :
====================================

[root@apps rpms]# groupadd dba
[root@apps rpms]# useradd -g dba applprod

[root@apps rpms]# ls -ltr
total 7884
-rwxr-xr-x 1 root root 1014759 Jan  3 17:33 openmotif21-2.1.30-11.RHEL4.6.i386.rpm
-rwxr-xr-x 1 root root    3692 Jan  3 17:33 compat-oracle-rhel4-1.0-5.i386.rpm
-rw-r--r-- 1 root root 3863540 Jan  3 17:33 compat-gcc-32-c++-3.2.3-47.3.i386.rpm
-rw-r--r-- 1 root root 3172139 Jan  3 17:33 compat-gcc-32-3.2.3-47.3.i386.rpm
[root@apps rpms]# rpm -ivh openmotif21-2.1.30-11.RHEL4.6.i386.rpm
warning: openmotif21-2.1.30-11.RHEL4.6.i386.rpm: Header V3 DSA signature: NOKEY, key ID b38a8516
Preparing...                ########################################### [100%]
   1:openmotif21            ########################################### [100%]
[root@apps rpms]# rpm -ivh compat-oracle-rhel4-1.0-5.i386.rpm
error: Failed dependencies:
        compat-gcc-32 is needed by compat-oracle-rhel4-1.0-5.i386
        compat-gcc-32-c++ is needed by compat-oracle-rhel4-1.0-5.i386
[root@apps rpms]# rpm -ivh compat-gcc-32-3.2.3-47.3.i386.rpm
warning: compat-gcc-32-3.2.3-47.3.i386.rpm: Header V3 DSA signature: NOKEY, key ID b38a8516
Preparing...                ########################################### [100%]
   1:compat-gcc-32          ########################################### [100%]
[root@apps rpms]# rpm -ivh compat-gcc-32-c++-3.2.3-47.3.i386.rpm
warning: compat-gcc-32-c++-3.2.3-47.3.i386.rpm: Header V3 DSA signature: NOKEY, key ID b38a8516
Preparing...                ########################################### [100%]
   1:compat-gcc-32-c++      ########################################### [100%]
[root@apps rpms]# rpm -ivh compat-oracle-rhel4-1.0-5.i386.rpm
Preparing...                ########################################### [100%]
   1:compat-oracle-rhel4    ########################################### [100%]
[root@apps rpms]#

3. Create the Directories for Target AppsTier :
================================

[root@apps a01]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              20G  5.0G   14G  28% /
/dev/sda3              70G  188M   66G   1% /a01
tmpfs                 2.0G     0  2.0G   0% /dev/shm
[root@apps a01]# cd /a01/
[root@apps a01]# ls -ltr
total 20
drwx------ 2 root root 16384 Jan  2 21:44 lost+found
drwxr-xr-x 2 root root  4096 Jan  3 17:33 rpms
[root@apps a01]# mkdir applprod
[root@apps a01]# ls -ltr
total 24
drwx------ 2 root root 16384 Jan  2 21:44 lost+found
drwxr-xr-x 2 root root  4096 Jan  3 17:33 rpms
drwxr-xr-x 2 root root  4096 Jan  3 17:38 applprod
[root@apps a01]# chown -R applprod:dba applprod/
[root@apps a01]# chmod -R 777 applprod/
[root@apps a01]#

4. Adcfgclone in AppsTier :
===================

[applprod@apps bin]$ cd /a01/applprod/prodcomn/clone/bin/
[applprod@apps bin]$ ls -ltr
total 164
-rwxrwxrwx 1 applprod dba 80663 Oct 28 16:09 adclone.pl
-rwxrwxrwx 1 applprod dba 29224 Oct 28 16:09 adclonectx.pl
-rwxrwxrwx 1 applprod dba  1226 Oct 28 16:09 adchkutl.sh
-rwxrwxrwx 1 applprod dba 33888 Oct 28 16:09 adcfgclone.pl
-rwxrwxrwx 1 applprod dba  7396 Jan  3 16:43 adaddnode.pl

*********
Error 1 :
*********

[applprod@apps bin]$ perl adcfgclone.pl appsTier
Enter the APPS password [APPS]:
apps
ERROR : Unable to set CLASSPATH
    /a01/applprod/prodcomn/clone/bin/../jlib/classes111.zip: is  missing

First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:

ERROR: context creation not completed successfully.
Please check /tmp/adcfgclone.err file for errors

[applprod@apps bin]$ vi /tmp/adcfgclone.err
[applprod@apps bin]$ cat /tmp/adcfgclone.err
/a01/applprod/prodcomn/clone/jre/1.3.1/bin/i386/native_threads/java: error while loading shared libraries: libstdc++-libc6.1-1.so.2: cannot open shared object file: No such file or directory
[applprod@apps bin]$

***********
Solution :
***********

login as: root
Access denied
root@192.168.1.62's password:
[root@apps ~]# cd /usr/lib
[root@apps lib]# ls -ltr libstdc++-libc6.2-2.so.3
lrwxrwxrwx 1 root root 31 Jan  2 21:53 libstdc++-libc6.2-2.so.3 -> libstdc++-3-libc6.2-2-2.10.0.so
[root@apps lib]# ln -s libstdc++-libc6.2-2.so.3 libstdc++-libc6.1-1.so.2
[root@apps lib]#

*********
Error 2 :
*********

Enter the port pool number [0-99]:
4

Checking the port pool 4
done: Port Pool 4 is free
Web Listener port is 8004
Complete port information available at /a01/applprod/prodappl/admin/out/prod_apps/portpool.lst

Creating the new APPL_TOP Context file from :
  /a01/applprod/prodappl/ad/11.5.0/admin/template/adxmlctx.tmp

The new APPL_TOP context file has been created :
  /a01/applprod/prodappl/admin/prod_apps.xml

Log file located at /a01/applprod/prodcomn/clone/bin/CloneContext_01030927.log

Running Rapid Clone with command:
        perl /a01/applprod/prodcomn/clone/bin/adclone.pl java=/a01/applprod/prodcomn/clone/bin/../jre/1.3.1 mode=apply stage=/a01/applprod/prodcomn/clone/bin/.. component=appsTier method=CUSTOM appctxtg=/a01/applprod/prodappl/admin/prod_apps.xml  showProgress contextValidated=true

Rapid Clone will use the oraInst.loc file.
The file oraInst.loc needs to exist at:
        /etc/oraInst.loc
The 806 installer uses the oratab file.
The file oratab needs to exist at:
        /etc/oratab
Please run the script /tmp/orainstRoot.sh as root

Restart adclone.pl after performing required corrections.

 ERROR: Failed to execute /a01/applprod/prodcomn/clone/bin/adclone.pl

 Please check logfile.
[applprod@apps bin]$

***********
Solution :
***********

1. [root@apps tmp]# su - applprod
[applprod@apps ~]$ cd /tmp/
[applprod@apps tmp]$ ls -ltr
total 40
drwx--x--x 17 root     root 4096 Jan  2 21:58 cim_schema.21774
drwx------  2 root     root 4096 Jan  3 14:26 ssh-luzqod3349
drwx------  2 root     root 4096 Jan  3 14:26 keyring-uZaf57
drwx------  3 root     root 4096 Jan  3 14:26 gconfd-root
drwx------  2 root     root 4096 Jan  3 14:26 virtual-root.wiw9WM
srwxr-xr-x  1 root     root    0 Jan  3 14:26 mapping-root
drwx------  2 root     root 4096 Jan  3 14:26 orbit-root
drwxr-xr-x  2 applprod dba  4096 Jan  3 21:29 hsperfdata_applprod
-rwxrwxrwx  1 applprod dba   256 Jan  3 21:29 orainstRoot.sh
[applprod@apps tmp]$ sh orainstRoot.sh
touch: cannot touch `/etc/oraInst.loc': Permission denied
orainstRoot.sh: line 4: /etc/oraInst.loc: Permission denied
touch: cannot touch `/etc/oratab': Permission denied
chmod: cannot access `/etc/oratab': No such file or directory
[applprod@apps tmp]$

2. [root@apps tmp]# cd /
[root@apps /]# chmod -R 777 etc
[root@apps /]# su - applprod
[applprod@apps ~]$ cd /tmp/
[applprod@apps tmp]$ sh orainstRoot.sh
[applprod@apps tmp]$

*********
Error 3 :
*********

Log file located at /home/applprod/oraInventory/logs/OracleHomeCloner_01030936.log
ORACLE_HOME /a01/applprod/prodora/iAS was registered successfully.

Starting application Services for prod:
Running:
 /a01/applprod/prodcomn/admin/scripts/prod_apps/adstrtal.sh APPS/<appspwd>

You are running adstrtal.sh version 115.14

/a01/applprod/prodcomn/admin/scripts/prod_apps/adstrtal.sh: line 122: /a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/bin/jre: No such file or directory
Check logfile /a01/applprod/prodcomn/admin/log/prod_apps/01032136.log for details

Exiting with status 127

****************
Solution Tried :
****************
 
[applprod@apps bin]$ ln -s /a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/bin/java /a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/bin/jre
ln: creating symbolic link `/a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/bin/jre' to `/a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/bin/java': File exists

5. Successful Adcfgclone.pl in Target APPSTier :
=================================

***********
Solution :
***********

[applprod@apps bin]$ pwd
/a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/bin
[applprod@apps bin]$ ln -s java jre

[applprod@apps bin]$ cd /a01/applprod/prodcomn/clone/bin/
[applprod@apps bin]$ ls -ltr
total 332
-rwxrwxrwx 1 applprod dba 80663 Oct 28 16:09 adclone.pl
-rwxrwxrwx 1 applprod dba 29224 Oct 28 16:09 adclonectx.pl
-rwxrwxrwx 1 applprod dba  1226 Oct 28 16:09 adchkutl.sh
-rwxrwxrwx 1 applprod dba 33888 Oct 28 16:09 adcfgclone.pl
-rwxrwxrwx 1 applprod dba  7396 Jan  3 16:43 adaddnode.pl
-rw-r--r-- 1 applprod dba 25546 Jan  3 21:29 CloneContext_01030927.log
-rw-r--r-- 1 applprod dba 25546 Jan  3 21:33 CloneContext_01030932.log
-rw-r--r-- 1 applprod dba 25546 Jan  3 21:47 CloneContext_01030946.log
-rw-r--r-- 1 applprod dba 78960 Jan  3 21:49 sqlnet.log
[applprod@apps bin]$ perl adcfgclone.pl appsTier
Enter the APPS password [APPS]:
apps

First Creating a new context file for the cloned system.
The program is going to ask you for information about the new system:


Provide the values required for creation of the new APPL_TOP Context file.

Do you want to use a virtual hostname for the target node (y/n) [n] ?:

Target system database SID [cln]:prod

Target system domain name [sapcle.com]:

Target system database server node [Test]:db

Does the target system have more than one application tier server node (y/n) [n] ?:

Is the target system APPL_TOP divided into multiple mount points (y/n) [n] ?:

Target system APPL_TOP mount point [/test/applmgr/uatappl]:/a01/applprod/prodappl

Target system COMMON_TOP directory [/test/applmgr/uatcomn]:/a01/applprod/prodcomn

Target system 8.0.6 ORACLE_HOME directory [/test/applmgr/uatora/8.0.6]:/a01/applprod/prodora/8.0.6

Target system iAS ORACLE_HOME directory [/test/applmgr/uatora/iAS]:/a01/applprod/prodora/iAS

Do you want to preserve the Display set to test:0.0 (y/n) [y] ?:n

Target system Display [apps:0.0]:apps:1.0

Do you want to preserve the port values from the source system on the target system (y/n) [y] ?:n

Clone Context uses the same port pool mechanism as the Rapid Install
Once you choose a port pool, Clone Context will validate the port availability.

Enter the port pool number [0-99]:
4

Checking the port pool 4
done: Port Pool 4 is free
Web Listener port is 8004
Complete port information available at /a01/applprod/prodappl/admin/out/prod_apps/portpool.lst

Backing up /a01/applprod/prodappl/admin/prod_apps.xml to /a01/applprod/prodappl/admin/prod_apps.xml1.bak

Creating the new APPL_TOP Context file from :
  /a01/applprod/prodappl/ad/11.5.0/admin/template/adxmlctx.tmp

The new APPL_TOP context file has been created :
  /a01/applprod/prodappl/admin/prod_apps.xml

Log file located at /a01/applprod/prodcomn/clone/bin/CloneContext_01040324.log

Running Rapid Clone with command:
        perl /a01/applprod/prodcomn/clone/bin/adclone.pl java=/a01/applprod/prodcomn/clone/bin/../jre/1.3.1 mode=apply stage=/a01/applprod/prodcomn/clone/bin/.. component=appsTier method=CUSTOM appctxtg=/a01/applprod/prodappl/admin/prod_apps.xml  showProgress contextValidated=true

Beginning application tier Apply - Sat Jan  4 15:25:54 2014
Log file located at /a01/applprod/prodappl/admin/prod_apps/log/ApplyAppsTier_01040325.log
  \     50% completed      
AC-50207: Fatal: Failed to execute one or more of the config tools during Context Value Management
Deactivating Context Value Management System.
Restored the Context file in :
        /a01/applprod/prodappl/admin/prod_apps.xml

Aborting the running of Context Value Management
Error occured before Context Value Management could be completed
  /     50% completed       WARNING: AutoConfig completed with errors. Check logfile at /a01/applprod/prodappl/admin/prod_apps/log/ApplyAppsTier_01040325.log for details.
                                         

Completed Apply...
Sat Jan  4 15:29:40 2014

Beginning APPSIAS_prod registration to central inventory...

ORACLE_HOME NAME   : APPSIAS_prod
ORACLE_HOME PATH   : /a01/applprod/prodora/iAS
INVENTORY LOCATION : /etc/oraInst.loc

Log file located at /home/applprod/oraInventory/logs/OracleHomeCloner_01040329.log
RC-00123: ORACLE_HOME APPSIAS_prod already exists inside the inventory

Starting application Services for prod:
Running:
 /a01/applprod/prodcomn/admin/scripts/prod_apps/adstrtal.sh APPS/<appspwd>

You are running adstrtal.sh version 115.14

Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adapcctl.sh start
script returned:
****************************************************

adapcctl.sh version 115.43

Starting Apache Web Server Listener (dedicated HTTP) ...
Starting Apache Web Server Listener (dedicated PLSQL) ...

adapcctl.sh: exiting with status 3


.end std out.

.end err out.

****************************************************


Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adalnctl.sh start
script returned:
****************************************************

adalnctl.sh version

Checking for FNDFS executable.
Starting listener process APPS_prod.

adalnctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adcmctl.sh start
script returned:
****************************************************

You are running adcmctl.sh version 115.19

Starting concurrent manager for prod ...
Starting prod_0104@prod Internal Concurrent Manager
Default printer is noprint

adcmctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adfrmctl.sh start
script returned:
****************************************************

You are running adfrmctl.sh version 115.33

Starting forms server for prod on port 9004.

adfrmctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adrepctl.sh start
script returned:
****************************************************

You are running adrepctl.sh version 115.26

starting Reports Server for prod on port 7004.

adrepctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adfmcctl.sh start
script returned:
****************************************************

You are running adfmcctl.sh version 115.16

Starting forms load balancing client for prod.

adfmcctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adfmsctl.sh start
script returned:
****************************************************

You are running adfmsctl.sh version 115.12

starting forms  metrics server for prod.

adfmsctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/jtffmctl.sh start
script returned:
****************************************************

You are running jtffmctl.sh version 115.16

Validating Fulfillment patch level via /a01/applprod/prodcomn/java
Fulfillment patch level validated.
Starting Fulfillment Server for prod on port 9304 ...

jtffmctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/addisctl.sh start
script returned:
****************************************************

addisctl.sh version 115.14

/a01/applprod/prodora/8.0.6/vbroker/bin/osagent
Started osagent.
Osagent logs messages to the file /a01/applprod/prodora/8.0.6/discwb4/util/osagent.log.
Waiting for OAD to start...
Started OAD.
OAD logs messages to the file /a01/applprod/prodora/8.0.6/discwb4/util/oad.log.
Failed to start locator.
Cannot bind to OAD. Re-starting...

OAD is stopped
Osagent is stopped
Unable to stop locator. No process-id  file found.
Unable to stop gatekeeper. No process-id  file found.
Unable to stop any Discoverer Sessions. No process-id  file found.
Unable to stop any Discoverer Preference. No process-id  file found.
Unable to stop any Discoverer Collector. No process-id  file found.
/a01/applprod/prodora/8.0.6/vbroker/bin/osagent
Started osagent.
Osagent logs messages to the file /a01/applprod/prodora/8.0.6/discwb4/util/osagent.log.
Waiting for OAD to start...
Started OAD.
OAD logs messages to the file /a01/applprod/prodora/8.0.6/discwb4/util/oad.log.
Failed to start locator.
Registering Discoverer Session
Registering the Collector
Applying preferences from file : /a01/applprod/prodora/8.0.6/discwb4/util/pref.txt
Finished applying preferences
Registering Discoverer Preference Repository

addisctl.sh: exiting with status 1


.end std out.
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
./applypreferences.sh: line 26: 31353: Memory fault
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM

.end err out.

****************************************************


Check logfile /a01/applprod/prodcomn/admin/log/prod_apps/01041529.log for details

Exiting with status 2
[applprod@apps bin]$

=================================================================

6. Run Autoconfig manually in AppsTier :
=============================

[applprod@apps prod_apps]$ sh adautocfg.sh
/a01/applprod/prodora/iAS/Apache/perl/bin/perl: error while loading shared libraries: libdb.so.3: cannot open shared object file: No such file or directory
[applprod@apps prod_apps]$

***********
Solution :
***********

Refernce Note id : EBS 11.5.10.2 Install Fails on OEL5 with AC-50207 and java.lang.Exception: Fatal Exception (Doc ID 948628.1)
 
[root@apps lib]# pwd
/usr/lib
[root@apps lib]# clear
[root@apps lib]# pwd
/usr/lib
[root@apps lib]# cd /a01/rpms/6078836/
[root@apps 6078836]# ls -ltr
total 16
-rw-r--r-- 1 root root 5825 Oct  4  2007 libdb.so.2
-rw-rw-r-- 1 root root 4819 Oct  4  2007 README.txt
[root@apps 6078836]# cp libdb.so.2 /usr/lib
[root@apps 6078836]# cd /usr/lib
[root@apps lib]# ln -s ./libdb
libdb-4.3.a              libdb_cxx-4.3.a          libdb_cxx.so             libdb.so                 libdbus-glib-1.so
libdb-4.3.la             libdb_cxx-4.3.la         libdbi.so.0              libdb.so.2               libdbus-glib-1.so.2
libdb-4.3.so             libdb_cxx-4.3.so         libdbi.so.0.0.5          libdb.so.3_ori           libdbus-glib-1.so.2.1.0
[root@apps lib]# ln -s ./libdb.so.2 libdb.so.3

[root@apps lib]# su - applprod
[applprod@apps prod_apps]$ sh adautocfg.sh
Enter the APPS user password:
AutoConfig is configuring the Applications environment...

AutoConfig will consider the custom templates if present.
        Using APPL_TOP location     : /a01/applprod/prodappl
        Using java command          : /a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/bin/jre
        Java version                : 1.4.2_04
        Classpath                   : /a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/jre/lib/rt.jar:/a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/lib/dt.jar:/a01/applprod/prodcomn/util/java/1.4/j2sdk1.4.2_04/lib/tools.jar:/a01/applprod/prodcomn/java/appsborg2.zip:/a01/applprod/prodcomn/java

        Using Context file          : /a01/applprod/prodappl/admin/prod_apps.xml

Context Value Management will now update the Context file

        Updating Context file...COMPLETED

        Attempting upload of Context file and templates to database...COMPLETED

Configuring templates from all of the product tops...
               Configuring AD_TOP........COMPLETED
                Configuring FND_TOP.......COMPLETED
            Configuring ICX_TOP.......COMPLETED
            Configuring IEO_TOP.......COMPLETED
            Configuring ABM_TOP.......COMPLETED
            Configuring ECX_TOP.......COMPLETED
            Configuring BIS_TOP.......COMPLETED
            Configuring GL_TOP........COMPLETED
            Configuring AMS_TOP.......COMPLETED
            Configuring CCT_TOP.......COMPLETED
            Configuring WSH_TOP.......COMPLETED
            Configuring CLN_TOP.......COMPLETED
            Configuring OKE_TOP.......COMPLETED
            Configuring OKL_TOP.......COMPLETED
            Configuring OKS_TOP.......COMPLETED
            Configuring CSF_TOP.......COMPLETED
            Configuring XNC_TOP.......COMPLETED
            Configuring IGS_TOP.......COMPLETED
            Configuring IBY_TOP.......COMPLETED
            Configuring PA_TOP........COMPLETED
            Configuring JTF_TOP.......COMPLETED
            Configuring MWA_TOP.......COMPLETED
            Configuring CN_TOP........COMPLETED
            Configuring CSI_TOP.......COMPLETED
            Configuring WIP_TOP.......COMPLETED
            Configuring CSE_TOP.......COMPLETED
            Configuring EAM_TOP.......COMPLETED
            Configuring IMT_TOP.......COMPLETED
            Configuring FTE_TOP.......COMPLETED
            Configuring ONT_TOP.......COMPLETED
            Configuring AR_TOP........COMPLETED
            Configuring AHL_TOP.......COMPLETED
            Configuring OZF_TOP.......COMPLETED
            Configuring IES_TOP.......COMPLETED
            Configuring CSD_TOP.......COMPLETED
            Configuring IGC_TOP.......COMPLETED
   
    AutoConfig completed successfully.
    The log file for this session is located at: /a01/applprod/prodappl/admin/prod_apps/log/01091530/adconfig.log

[applprod@apps prod_apps]$ pwd
/a01/applprod/prodcomn/admin/scripts/prod_apps

7. Manually Bring up the Application Services :
=================================

[applprod@apps prod_apps]$ adstrtal.sh apps/apps

You are running adstrtal.sh version 115.14

Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adapcctl.sh start
script returned:
****************************************************

adapcctl.sh version 115.43

Starting Apache Web Server Listener (dedicated HTTP) ...
Starting Apache Web Server Listener (dedicated PLSQL) ...

adapcctl.sh: exiting with status 0
 
.end std out.

.end err out.

****************************************************

Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adalnctl.sh start
script returned:
****************************************************

adalnctl.sh version

Checking for FNDFS executable.
Starting listener process APPS_prod.

adalnctl.sh: exiting with status 0
 
.end std out.

.end err out.

****************************************************
 
Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adcmctl.sh start
script returned:
****************************************************

You are running adcmctl.sh version 115.19

Starting concurrent manager for prod ...
Starting prod_0109@prod Internal Concurrent Manager
Default printer is noprint

adcmctl.sh: exiting with status 0
 
.end std out.

.end err out.

****************************************************
 
Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adfrmctl.sh start
script returned:
****************************************************

You are running adfrmctl.sh version 115.33

Starting forms server for prod on port 9004.

adfrmctl.sh: exiting with status 0
 
.end std out.

.end err out.

****************************************************
 
Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adrepctl.sh start
script returned:
****************************************************

You are running adrepctl.sh version 115.26

starting Reports Server for prod on port 7004.

adrepctl.sh: exiting with status 0
 
.end std out.

.end err out.

****************************************************
 
Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adfmcctl.sh start
script returned:
****************************************************

You are running adfmcctl.sh version 115.16

Starting forms load balancing client for prod.

adfmcctl.sh: exiting with status 0
 
.end std out.

.end err out.

****************************************************

Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/adfmsctl.sh start
script returned:
****************************************************

You are running adfmsctl.sh version 115.12

starting forms  metrics server for prod.

adfmsctl.sh: exiting with status 0
 
.end std out.

.end err out.

****************************************************
 
Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/jtffmctl.sh start
script returned:
****************************************************

You are running jtffmctl.sh version 115.16

Validating Fulfillment patch level via /a01/applprod/prodcomn/java
Fulfillment patch level validated.
Starting Fulfillment Server for prod on port 9304 ...

jtffmctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


Executing service control script:
/a01/applprod/prodcomn/admin/scripts/prod_apps/addisctl.sh start
script returned:
****************************************************

addisctl.sh version 115.14

/a01/applprod/prodora/8.0.6/vbroker/bin/osagent
Started osagent.
Osagent logs messages to the file /a01/applprod/prodora/8.0.6/discwb4/util/osagent.log.
Waiting for OAD to start...
Started OAD.
OAD logs messages to the file /a01/applprod/prodora/8.0.6/discwb4/util/oad.log.
Failed to start locator.
Cannot bind to OAD. Re-starting...

OAD is stopped
Osagent is stopped
Unable to stop locator. No process-id  file found.
Unable to stop gatekeeper. No process-id  file found.
Unable to stop any Discoverer Sessions. No process-id  file found.
Unable to stop any Discoverer Preference. No process-id  file found.
Unable to stop any Discoverer Collector. No process-id  file found.
/a01/applprod/prodora/8.0.6/vbroker/bin/osagent
Started osagent.
Osagent logs messages to the file /a01/applprod/prodora/8.0.6/discwb4/util/osagent.log.
Waiting for OAD to start...
Started OAD.
OAD logs messages to the file /a01/applprod/prodora/8.0.6/discwb4/util/oad.log.
Failed to start locator.
Registering Discoverer Session
Registering the Collector
Applying preferences from file : /a01/applprod/prodora/8.0.6/discwb4/util/pref.txt
Finished applying preferences
Registering Discoverer Preference Repository

addisctl.sh: exiting with status 1


.end std out.
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM
./applypreferences.sh: line 26: 18570: Memory fault
/a01/applprod/prodora/8.0.6/jre1183o/lib/i686/green_threads/libzip.so: symbol errno, version GLIBC_2.0 not defined in file libc.so.6 with link time reference (libzip.so)
Unable to initialize threads: cannot find class java/lang/Thread
Could not create Java VM

.end err out.

****************************************************


Check logfile /a01/applprod/prodcomn/admin/log/prod_apps/01091535.log for details

Exiting with status 1

[applprod@apps prod_apps]$ adapcctl.sh status

adapcctl.sh version 115.43

Apache Web Server Listener :httpd ( pid 17835 ) is running.
Apache Web Server Listener :httpd ( pid 17919 ) is running.

adapcctl.sh: exiting with status 0

[applprod@apps prod_apps]$ adcmctl.sh status apps/apps

You are running adcmctl.sh version 115.19
 
adcmctl.sh: exiting with status 0

[applprod@apps prod_apps]$ ps -ef|grep FNDLIBR
applprod 18069 18064  0 15:35 pts/1    00:00:00 FNDLIBR
applprod 18659 18609  0 15:36 ?        00:00:00 FNDLIBR                                                                                                                
applprod 18660 18609  0 15:36 ?        00:00:00 FNDLIBR                                                                                                                
applprod 18661 18609  0 15:36 ?        00:00:00 FNDLIBR                                                                                                                
applprod 18671 18609  0 15:36 ?        00:00:00 FNDLIBR                                                                                                                
applprod 18687 18609  0 15:36 ?        00:00:00 FNDLIBR                                                                                                                
applprod 19098 11023  0 15:37 pts/1    00:00:00 grep FNDLIBR
[applprod@apps prod_apps]$

===============================================================================================================================================


8. Active Users Report completed with error:
===============================

report Log file error :
===============

ar60run: relocation error: /a01/applprod/prodora/8.0.6/network/jre11/lib/i686/native_threads/libjava.so: symbol __libc_wait, version GLIBC_2.0 not defined in file libc.so.6 with link time reference

Program exited with status 127
Concurrent Manager encountered an error while running Oracle*Report for your concurrent request 2721374.

Review your concurrent request log and/or report output file for more detailed information.

**********
Solution :
**********

1. Apply the rpm compat-libcwait-2.1-1.i386.rpm

or

2. Put the following line in .bash_profile in root as well as oracle :

LD_ASSUME_KERNEL=2.4.19
export LD_ASSUME_KERNEL

***************************************************************
Note : After the RPM installation report completed with normal.
***************************************************************

Regards,
Vijay.