Oracle Fusion Applications Installation: Configuring Oracle Identity and Access Management components

Previous: Installing Oracle Identity and Access Management Components

Configuring Oracle Identity Management components” can be divided into following tasks.

1. Configuring the Web Tier
2. Create Weblogic Domain for Identity Management
3. Extending the Domain with Oracle Internet Directory
4. Extending the Domain with Oracle Directory Service Manager (ODSM)
5. Extending the Domain with Oracle Virtual Directory
6. Extending the Domain with Oracle Access Manager
7. Extending the Domain to Configure Oracle Identity Manager and Oracle SOA Suite
8. Installing and Configuring WebGate

Please note that this post is going to be long so allow time to load all images.

Configuring the Web Tier

Start the configuration from <Web_Home>/bin

[oracle@fusion bin]$ cd /app/fusion/bea_default/Oracle_WT1/bin

[oracle@fusion bin]$ ./config.sh

Click Next

Select “Oracle HTTP Server” and click Next

For Instance location enter “/app/fusion/bea_default/instances” since we will keep all instances in this location. Provide any appropriate Instance name and OHS component name. We will go for the defaults. Click Next

Select “Specify Ports using Configuration File” and enter file name as /home/oracle/staticports.ini

Now we will copy staticports.ini default file from <repository_location>/installers/webtier/Disk1/stage/Response to home directory /home/oracle

cp /mnt/fusion/installers/webtier/Disk1/stage/Response/staticports.ini ~/staticports.ini

Now click on “View/Edit File” to edit this file.

Uncomment and set the following values. Click Save

OPMN Local Port = 6700

OHS Port = 7777

Click Next

Deselect email notification and click Next

Save summary if needed and click Configure to start configuration.

Important Note: If SELinux is enabled in your Linux operating System then it will throw an error. Since we already disabled it during installation, we will not see that error here.

Once installation finishes, click Next

Save installation summary if needed and click Finish to complete the installation.

It would have already started HTTP server now. We can verify the same.

[oracle@fusion instances]$ ps -ef | grep http

oracle 3521 3491 0 10:06 ? 00:00:00 /app/fusion/bea_default/Oracle_WT1/ohs/bin/httpd.worker -DSSL

oracle 3547 3521 0 10:06 ? 00:00:00 /app/fusion/bea_default/Oracle_WT1/ohs/bin/httpd.worker -DSSL

oracle 3548 3521 0 10:06 ? 00:00:00 /app/fusion/bea_default/Oracle_WT1/ohs/bin/httpd.worker -DSSL

oracle 3549 3521 0 10:06 ? 00:00:00 /app/fusion/bea_default/Oracle_WT1/ohs/bin/httpd.worker -DSSL

Check /app/fusion/bea_default/Oracle_WT1/instances/instance1/config/OHS/ohs1/httpd.conf to make sure it reflects correct user and group name

User oracle

Group oinstall

We can launch http://fusion:7777 (Homepage of Oracle HTTP server) now. It will look as follows.

Create Weblogic Domain for Identity Management

Start the configuration from <Middleware Home>/oracle_common/commin/bin

[oracle@fusion bin]$ /app/fusion/bea_default/oracle_common/common/bin/config.sh

Select “Create a new Weblogic domain” and click Next

Select “Oracle Enterprise Manager – 11.1.1.0 [oracle_common]” and “Oracle JRF – 11.1.1.0 [oracle_common]” and click Next

Enter details as above or accept default values and click Next.

Since it accepts minimum 8 characters set password again to
oracle123. Please note that you can also change username from weblogic but in future whenever we refer to weblogic user you must enter the new user which selected. We will go for default “weblogic” username

You would see option of Oracle JRockit here. So select that JDK in this list.

Select “Administration Server” and “Managed Servers, Clusters and Machines”. Click Next

Keep defaults but make a note of the port since this will be widely used during next part of installation. Click Next

Just click Next

Click Next again

Select Second Tab “Unix Machine” and enter the hostname as above. Click Next

Click on AdminServer and Click right arrow. Click Next

It will now look as above. Click Next

On Summary page click Create

Once installation finishes, click Done

Make sure that the encrypted username and password values are already in boot.properties

[oracle@fusion security]$ more /app/fusion/bea_default/user_projects/domains/IDM_domain/servers/AdminServer/security/boot.properties

# Generated by Configuration Wizard on Mon Jan 23 10:59:07 GST 2012

username={AES}zaXc3+4y2KGuxnK6WkI7ehKcliQDeandkjdTdu0vpuY=

password={AES}WZ6Zo+j6aGoCyE2nQmCCdboEkA8TDGRlagdSqFGRedo=

Set StartScriptEnabled=true in nodemanager.properties by running following script

[oracle@fusion bin]$ cd /app/fusion/bea_default/oracle_common/common/bin

[oracle@fusion bin]$ ./setNMProps.sh

Appending required nodemanager.properties

Verify the change.

[oracle@fusion bin]$ tail -f /app/fusion/bea_default/wlserver_10.3/common/nodemanager/nodemanager.properties

#Required NM Property overrides (append to existing nodemanager.properties)

StartScriptEnabled=true

Start Node manager

[oracle@fusion bin]$ cd /app/fusion/bea_default/wlserver_10.3/server/bin

[oracle@fusion bin]$ nohup ./startNodeManager.sh &

…

IDM_domain -> /app/fusion/bea_default/user_projects/domains/IDM_domain

…

INFO: Secure socket listener started on port 5556

Start Weblogic AdminServer

[oracle@fusion bin]$ nohup
/app/fusion/bea_default/user_projects/domains/IDM_domain/bin/startWebLogic.sh &

tail nohup.out file until it shows following message.

<Jan 23, 2012 11:55:21 AM GST> <Notice> <WebLogicServer> <BEA-000360> <Server started in RUNNING mode>

Open Weblogic Admin Console

Launch Weblogic Admin console through
http://fusion:7001/console

Login with weblogic/oracle123

Note: Go to preferences and change “automatic acquire lock” settings to avoid accidental changes since we are in development mode.

Configuring HTTP server for the Administration Server

Create a new file admin.conf as follows.

[oracle@fusion moduleconf]$ more /app/fusion/bea_default/Oracle_WT1/instances/instance1/config/OHS/ohs1/moduleconf/admin.conf

<VirtualHost *:7777>

ServerName fusion:80

ServerAdmin you@your.address

RewriteEngine On

RewriteOptions inherit

RewriteRule ^/console/jsp/common/logout.jsp /oamsso/logout.html [PT]

RewriteRule ^/em/targetauth/emaslogout.jsp /oamsso/logout.html [PT]

# Admin Server and EM

<Location /console>

SetHandler weblogic-handler

WebLogicHost fusion

WeblogicPort 7001

</Location>

<Location /consolehelp>

SetHandler weblogic-handler

WebLogicHost fusion

WeblogicPort 7001

</Location>

<Location /em>

SetHandler weblogic-handler

WebLogicHost fusion

WeblogicPort 7001

</Location>

</VirtualHost>

Restart http server.

ORACLE_HOME=/app/fusion/bea_default/Oracle_WT1

export ORACLE_HOME

ORACLE_INSTANCE=/app/fusion/bea_default/Oracle_WT1/instances/instance1

export ORACLE_INSTANCE

PATH=$ORACLE_HOME/opmn/bin:$PATH

export PATH

 

opmnctl stopall

opmnctl startall

Now we need to Register HTTP server with Weblogic Server so that Enterprise Manager can monitor the instance.

[oracle@fusion ~]$ opmnctl registerinstance -adminHost fusion -adminport 7001 -adminUsername weblogic

Command requires login to weblogic admin server (fusion):

Username: weblogic

Password:

…

Done

Registering instance

Command succeeded.

Note: We will not enable load-balancer access since we have skipped load-balancing in this single node installation guide.

Enable Weblogic Plugin

Log in the Oracle Weblogic Server Administration and click on Lock and Edit. Click on IDMDomain and Click on Configuration tab and then select the Web Applications tab.

Scroll down and enable “Weblogic Plug-in Enabled”.

Click on Save and Activate the Changes.

Restart the Weblogic Administration Server.

Extending the Domain with Oracle Internet Directory

Start the configuration from <IDM Oracle Home/bin

[oracle@fusion bin]$ cd /app/fusion/bea_default/Oracle_IDM1/bin

[oracle@fusion bin]$ ./config.sh &

Click Next

Select “Configure Without A Domain” and click Next

Make sure to select any directory inside MW_HOME/instances. You can accept default for Instance Name. Click Next

Deselect email notification and click Next

Select only “Oracle Internet Directory” and click Next

Select “Specify Ports using Configuration File” and enter file name as /home/oracle/staticports.ini

In another terminal window copy the staticports.ini file to home directory.

[oracle@fusion bin]$ cp /app/fusion/provisioning/idm/idm/Disk1/stage/Response/staticports.ini ~/

Click on View/Edit file

VERY IMPORTANT:

As per Oracle Manual we should Change it to as follows.

#The Non-SSL port for OID

Oracle Internet Directory Port No = 389

#The SSL port for OID

Oracle Internet Directory (SSL) Port No = 636

 

But OID fails to configure and start at the end of installation with these values so we will stick to the OID values for 11g in the staticports.ini and just remove the comments.

 

#The Non-SSL port for OID

Oracle Internet Directory Port No = 3060

#The SSL port for OID

Oracle Internet Directory (SSL) Port No = 3061

Click Save

Once saved, click Next

Enter oracle123 or any suitable password. If you are using different passwords then please make a note of all of them. Click Next

Since we are not using any domains as such but as we have added an entry in our hosts file for fusion.localdomain, we will add “dc=localdomain” for Realm. Enter oracle123 or any suitable password. Click Next

Save summary if needed and click Configure to start configuration.

Once installation finishes, click Next

Save installation summary if needed and click Finish to complete the installation.

Validate the OID installation

[oracle@fusion ~]$ export ORACLE_HOME=/app/fusion/bea_default/Oracle_IDM1

[oracle@fusion ~]$ export ORACLE_INSTANCE=/app/fusion/bea_default/instances/oid_inst1

[oracle@fusion ~]$ export PATH=$ORACLE_HOME/opmn/bin:$ORACLE_HOME/bin:$ORACLE_HOME/ldap/bin:$ORACLE_HOME/ldap/admin:$PATH

[oracle@fusion ~]$ ldapbind -h fusion -p 3060 -D “cn=orcladmin” -q

Please enter bind password:

bind successful

[oracle@fusion ~]$ ldapbind -h fusion -p 3061 -D “cn=orcladmin” -q -U 1

Please enter bind password:

bind successful

[oracle@fusion ~]$ opmnctl status

Processes in Instance: oid_inst1

———————————+——————–+———+———

ias-component | process-type | pid | status

———————————+——————–+———+———

oid1 | oidldapd | 19810 | Alive

oid1 | oidldapd | 19798 | Alive

oid1 | oidmon | 19785 | Alive

EMAGENT | EMAGENT | 19325 | Alive

Registering Oracle Internet Directory with the WebLogic Server Domain

[oracle@fusion ~]$ echo $ORACLE_HOME

/app/fusion/bea_default/Oracle_IDM1

[oracle@fusion ~]$ echo $ORACLE_INSTANCE

/app/fusion/bea_default/instances/oid_inst1

[oracle@fusion ~]$ opmnctl registerinstance -adminHost fusion -adminPort 7001 -adminUsername weblogic

Command requires login to weblogic admin server (fusion):

Username: weblogic

Password:

Registering instance

Command succeeded.

Note: We have skipped next steps related to SSL since we are setting up non-SSL connections here.

Update the Enterprise Manager Repository URL

Next we will update the Enterprise Manager Repository URL using the emctl utility with the switchOMS flag. The emctl utility is located under the ORACLE_INSTANCE/EMAGENT/EMAGENT/bin directory.

[oracle@fusion ~]$ cd /app/fusion/bea_default/instances/oid_inst1/EMAGENT/EMAGENT/bin

[oracle@fusion bin]$ ./emctl switchOMS http://fusion:7001/em/upload

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.

SwitchOMS succeeded.

We can now verify whether this instance is registered for monitoring agent.

Login to http://fusion:7001/em

Click on Farm->Agent monitored targets.

You should be able to see the OID instance under this. Make sure that the Agent URL is showing up fine. If the status shows as “Need Configuration” then click on Configure and then in next screen enter weblogic user credentials. It should now show the correct Agent URL.

Extending the Domain with Oracle Directory Service Manager (ODSM)

Start the configuration from <IDM Oracle Home>/bin

[oracle@fusion bin]$ cd /app/fusion/bea_default/Oracle_IDM1/bin

[oracle@fusion bin]$ ./config.sh &

Click Next

Select “Extend Existing Domain” and enter details of existing weblogic Server and AdminServer port. Click Next

You can ignore this error since we created this domain using the Identity Management installer. Click Yes to ignore.

Verify that the weblogic server directory shown is correct as per previous steps. Specify name and path for Oracle Directory Service instance. Make sure to keep the instance in same parent directory as previous instances. Click Next

Deselect email notification and click Next

Select Oracle “Directory Service Manager” and click Next

Select “Specify Ports using Configuration File” and enter file name as /home/oracle/staticports.ini

Meanwhile in another terminal window copy the staticports.ini to home directory.

[oracle@fusion bin]$ cp -p /app/fusion/provisioning/idm/idm/Disk1/stage/Response/staticports.ini ~/

Uncomment the ODS Server Port and keep it default 7006. Click Save.

Once Saved click Next.

Save summary if needed and click Configure to start configuration.

Once installation finishes, click Next

Save installation summary if needed and click Finish to complete the installation.

Password-less startup for ODS

cd /app/fusion/bea_default/user_projects/domains/IDM_domain/servers/wls_ods1/security

cp ../../AdminServer/security/boot.properties .

cd /app/fusion/bea_default/user_projects/domains/IDM_domain/bin

nohup ./startManagedWebLogic.sh wls_ods1

Now you can access ODS homepage at http://fusion:7006/odsm/faces/odsm.jspx (as per the above post-installation summary screen)

Register ODS with OID

Click on Connect to a directory -> Create A New Connection

Enter the details for OID.

Name: fusion-oid

Server: fusion

SSL Enabled: Unchecked

User Name: cn=orcladmin

Password: oracel123

Start Page: Home

Click Connect

Once connection is successful, you should be able to see OID page.

You can randomly check whether you are able to see details of any user, for example cn=orcladmin

Configuring Oracle HTTP Servers to Access the ODSM Console

[oracle@fusion moduleconf]$ cd /app/fusion/bea_default/Oracle_WT1/instances/instance1/config/OHS/ohs1/moduleconf

[oracle@fusion moduleconf]$ vi admin.conf

# Append following lines in admin.conf

<Location /odsm>

SetHandler weblogic-handler

WebLogicCluster fusion:7006

</Location>

Restart HTTP server.

Now we can access ODSM through http://fusion:7777/odsm/faces/odsm.jspx

Extending the Domain with Oracle Virtual Directory

Start the configuration from <IDM Oracle Home>/bin

[oracle@fusion ~]$ cd /app/fusion/bea_default/Oracle_IDM1/bin/

[oracle@fusion bin]$ ./config.sh &

Click Next

Select “Configure Without A Domain“. Click Next

Provide values for Oracle Virtual Directory (OVD) instance. You can accept default values. Click Next

Deselect email notification and click Next

Select “Oracle Virtual Directory” and click Next

Select “Specify Ports using Configuration File” and enter file name as /home/oracle/staticports.ini

Open a separate terminal/putty window and copy staticports.ini file to home directory.

[oracle@fusion ~]$ cp /app/fusion/provisioning/idm/idm/Disk1/stage/Response/staticports.ini ~/

Click on View/Edit.

Edit the staticports.ini file to assign ports 6501 and 7501, as follows.

# The non-SSL port for Oracle Virtual Directory

Oracle Virtual Directory port = 6501

# The SSL port for Oracle Virtual Directory

Oracle Virtual Directory (SSL) port = 7501

Click Save and then Next

Provide OID login details.

Uncheck “Configure Administrative Server in secure mode” since we are not using SSL anywhere in this installation. Click Next

Click Ok

Save summary if needed and click Configure to start configuration.

Once installation finishes, click Next

Save installation summary if needed and click Finish to complete the installation.

Registering OVD with the Oracle WebLogic Server Domain

[oracle@fusion bin]$ cd /app/fusion/bea_default/instances/ovd_inst1/bin

[oracle@fusion bin]$ ./opmnctl registerinstance -adminHost fusion -adminPort 7001 -adminUsername weblogic

Command requires login to weblogic admin server (fusion):

Username: weblogic

Password:

Registering instance

Command succeeded.

Update the Enterprise Manager Repository URL using the emctl utility with the switchOMS flag.

The emctl utility is located under the ORACLE_INSTANCE/EMAGENT/EMAGENT/bin directory.

[oracle@fusion bin]$ cd /app/fusion/bea_default/instances/ovd_inst1/EMAGENT/EMAGENT/bin

[oracle@fusion bin]$ ./emctl switchOMS http://fusion:7001/em/upload

Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.

Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.

SwitchOMS succeeded.

We can now verify whether this instance is registered for monitoring agent.

Login to http://fusion:7001/em

Click on Farm->Agent monitored targets.

You should be able to see the OID and OVD instances under this. Make sure that the Agent URL is showing up fine. If the status shows as “Need Configuration” then click on Configure and then in next screen enter weblogic user credentials. It should now show the correct Agent URL.

Validate the Oracle Virtual Directory Instances

[oracle@fusion ~]$ export ORACLE_HOME=/app/fusion/bea_default/Oracle_IDM1

[oracle@fusion ~]$ export ORACLE_INSTANCE=/app/fusion/bea_default/instances/ovd_inst1

[oracle@fusion ~]$ export PATH=$ORACLE_HOME/opmn/bin:$ORACLE_HOME/bin:$ORACLE_HOME/ldap/bin:$ORACLE_HOME/ldap/admin:$PATH

[oracle@fusion bin]$ ldapbind -h fusion -p 6501 -D “cn=orcladmin” –q

Please enter bind password:

bind successful

[oracle@fusion bin]$ ldapbind -h fusion -p 7501 -D “cn=orcladmin” -q -U 1

Please enter bind password:

SSL handshake failed

This is fine since we did not configure SSL mode.

Creating ODSM connections to Oracle Virtual Directory

Open http://fusion:7777/odsm/faces/odsm.jspx

Click Connect to a directory -> Create a new connection

Enter details of OVD here.

Name: fusion-ovd

Host: fusion

Port: 8899

SSL Enabled: Unchecked

User Name: cn=orcladmin

Password: oracle123

Start Page: Home

Click Connect

It will display OVD home page.

Note: We are skipping the adapter creation for now. We will create them later once needed.

Extending the Domain with Oracle Access Manager

Start the configuration from <Middleware Home/oracle_common/commin/bin

[oracle@fusion bin]$ cd /app/fusion/bea_default/oracle_common/common/bin

[oracle@fusion bin]$ ./config.sh

Click “Extend an existing WebLogic domain” and click Next

Scroll down and select user_projects->domains->IDM_domain. Click Next

Select only “Oracle Access Manager with Database Policy Store” and click on Next.

Here you need to provide the database connection details and choose schema owner username. You can keep the name default but make sure to keep a note of it since you will need this later. Choose a password for example oracle123. Click Next

Now it will test the database connectivity through JDBC.

Once successful, click Next

Select “Managed Servers, Clusters and Machines” only and click Next

The first row was from previous configuration of ODS and now there will be another row for OAM. Keep the default port and make a note of it. Enter wls_oam1 for the instance name (or whichever you chose while creating instance) and click Next

Just click Next

On the “Machines” tab make sure that correct hostname is entered. Click Next

Select wls_oam1 and click the right arrow.

Now it should look as above. Click Next

On the summary page Click Extend.

Click OK Since we are aware that these are correct ports being used by AdminServer and wls_ods1 instance.

Once configuration finishes, click Done.

You can start the managed server by using following command.

[oracle@fusion bin]$ cd /app/fusion/bea_default/user_projects/domains/IDM_domain/bin

[oracle@fusion bin]$ ./startManagedWebLogic.sh wls_oam1

This will create the directory /app/fusion/bea_default/user_projects/domains/IDM_domain/servers/wls_oam1

Press CTRL+C to stop the process since we will need to configure startup without password prompt.

[oracle@fusion wls_oam1]$ cd /app/fusion/bea_default/user_projects/domains/IDM_domain/servers/wls_oam1

[oracle@fusion wls_oam1]$ cp ../wls_ods1/security/ boot.properties .

Now we can start the managed server without prompting for password.

[oracle@fusion bin]$ cd /app/fusion/bea_default/user_projects/domains/IDM_domain/bin

[oracle@fusion bin]$ nohup ./startManagedWebLogic.sh wls_oam1 &

Configuring Oracle HTTP Servers to Display Login Page and Oracle Access Manager Console

Append following entries in /app/fusion/bea_default/Oracle_WT1/instances/instance1/config/OHS/ohs1/moduleconf/admin.conf

 <Location /oam>

SetHandler weblogic-handler

WebLogicCluster fusion:14100

</Location>

 

<Location /oamconsole>

SetHandler weblogic-handler

WebLogicHost fusion

WebLogicPort 7001

</Location>

Restart HTTP Server to bring this to effect.

Note: There are few other steps as well as per Oracle documentation but we will skip them for now.

Extending the Domain to Configure Oracle Identity Manager and Oracle SOA Suite

Start the configuration from <Middleware Home/oracle_common/common/bin

[oracle@fusion bin]$ cd /app/fusion/bea_default/oracle_common/common/bin

[oracle@fusion bin]$ ./config.sh &

Select “Extend an existing WebLogic domain” and click Next

Scroll down and select user_projects->domains->IDM_domain and click Next

Select Oracle Identity Manager. It will automatically select Oracle SOA Suite and Oracle WSM. Click Next

Check all to modify all entries together. Provide database connect details and select a password for all. Accept default value for Schema owner names. Click Next

Now it will test the database connectivity through JDBC.

Once JDBC test is successful, click Next

Select only “Managed Servers, Clusters and Machines” and click Next

Add entries for soa_server1 and oim_server1. As per Oracle documentation you can change them to wls_soa1 and wls_oim1 as well. But here we have kept them as default.

Note the ports and click Next.

Just click Next

Since we are using Linux/Unix machine, delete entry from above screen. And proceed to Unix Machine tab.

Make sure correct hostname is entered here. Click Next

Select oim_server1 and soa_server1 and click right arrow. (if you had renamed them to wls_oim1 and wls_soa1 then you will see those entries instead of this)

The screen will now look like above. Click Next

On Summary screen click Next

Click OK

Once configuration finishes, click Done

Configuring Oracle Identity Manager

Now we will configure the Identity Manager from <IAM Oracle Home>/bin

[oracle@fusion bin]$ cd /app/fusion/bea_default/Oracle_IAM/bin

[oracle@fusion bin]$ ./config.sh &

Click Next

Select only OIM Server and click Next

Enter database details in shown format “fusion:1521:fusiondb“. Select Schema names (keep default) and enter password (oracle123). Make sure to keep a note of these schema names DEV_OIM and DEV_MDS. We will need these later during provisioning plan. Click Next

Enter AdminServer details in t3://<hostname>:<port> format. Here t3://fusion:7001

Important Note: Before clicking next make sure that AdminServer is running otherwise it may throw following error on next page. Start or restart AdminServer if you see this error.

INST-6180: Error while retrieving OIM Managed Server URL from the domain.

Click Next

Enter passwords as follows and keep a note of them since we will require them in provisioning wizard.

OIM Admin password: Oracle123

Keystore Password: oracle123

Enter OIM HTTP URL as http://fusion:14000 (based on port value in previous configuration step). Click Next

Deselect both and click Next

Save the summary if required and click Configure.

Once configuration finishes click Next

Save the configuration summary if needed and click Finish to complete the configuration.

Installing and Configuring WebGate

Start Webgate 10g installation from <repository_location>/installers/webgate

[oracle@fusion webgate]$ cd /mnt/fusion/installers/webgate

[oracle@fusion webgate]$ ./Oracle_Access_Manager10_1_4_3_0_linux64_OHS11g_WebGate –gui

Click Next

Enter the OS oracle user details. Click Next

Select a path where you want to install webgate. We have selected /app/fusion/webgate. Click Next

Click Next

Open a new terminal/putty window and execute following commands. Once done enter the location /app/fusion/oam_lib in this screen and click Next to start the installation.

[oracle@fusion oam_lib]$ mkdir /app/fusion/oam_lib

[oracle@fusion oam_lib]$ ln -s /usr/lib64/libstdc++.so.5 /app/fusion/oam_lib/libstdc++.so.5

[oracle@fusion oam_lib]$ ln -s /lib64/libgcc_s.so.1 /app/fusion/oam_lib/libgcc_s.so.1

Now WebGate Configuration will start. Select “Open Mode” and click Next

Since this screen requires WebGate ID from Access Manager. We need to first create a Webgate Agent in Access Manager. So keep this Java window open and open Oracle Access Manager Console in browser by entering http://fusion/oamconsole or http://fusion:7777/oamconsole

Note: We could have done this before starting Webgate installation as well but keeping it here in the guide gives you an idea where we are going to this configuration. Next time when you install, you can do this step before itself.

Login with weblogic admin user.

Once logged under SSO Agents, click on New “OAM 10g Webgate” (since we are installing 10g webgate).

Enter name: Webgate_sso or any name

Base URL: http://fusion:7777

Host Identifier: Webgate_sso

Security: Open

Uncheck all 3 options on right.

Click Apply

Now Edit the same Webgate Agent again. It will show following screen.

We did not put domain since we are using only http://fusion

Change Max Connections to 4

Logout URL: Enter following values.

/oamconsole/logout.html

/console/jsp/common/logout.jsp

/em/targetauth/eamlogout.jsp

Click Apply

Once webgate Agent is created in OAM, go back to the already open configuration window and enter following details and click Next to finish the configuration. In case if the configuration window was closed, you can also launch the configuration from command prompt.

[oracle@fusion webgate]$ /app/fusion/webgate/access/oblix/tools/configureWebGate/start_configureWebGate -i /app/fusion/webgate/access -t WebGate

Please enter the Mode in which you want the Web Gate to run : 1(Open) 2(Simple) 3(Cert) : 1

Please enter the Web Gate ID : Webgate_sso

Please enter the Password for this Web Gate : <enter oracle123 or any password here. Make note of it>

Please enter the Access Server ID : wls_oam1

Please enter the Access Server Host Machine Name : fusion

Please enter the Access Server Port : 5575

Preparing to connect to Access Server. Please wait.

Web Gate installed Successfully.

Press enter key to continue …

As per Oracle documentation there are a few steps to use the Webgate and validate but we will skip them for now.

Next step is to prepare Identity and Policy Stores by creating necessary users and groups for provisioning Fusion Applications.

Next: Preparing Identity and Policy Stores

Installing Oracle Fusion Applications – steps

1. Installing Fusion Applications Provisioning Framework
   
2. Installing Oracle 11g Database (Applications Transactional Database)
   
3. Running Oracle Fusion Applications Repository Creation Utility (Applications RCU)
   
4. Creating another database for Oracle  Identity Management Infrastructure (optional)
   
5. Running Repository Creation Utility (RCU) for Oracle Identity Management components
6. Installing Oracle Identity and Access Management Components
   
7. Configuring Oracle Identity and Access Management components
   
8. Preparing Identity and Policy Stores
   
9. Creating a New Provisioning Plan
   
10. Provisioning an Applications Environment

Oracle Fusion Applications Installation: Creating a new Provisioning Plan

Previous: Preparing Identity and Policy Store

In order to create a new fusion applications provisioning plan we should launch the provisioning wizard again from <framework_location>/provisioning/bin

<framework_location> is same what we mentioned in previous post. i.e. /app/fusion

[oracle@fusion $ cd /app/fusion/provisioning/bin

[oracle@fusion bin]$ ./provisioningWizard.sh

On Welcome screen, click Next

Select “Create a New Applications Environment Provisioning Plan” and click Next

Deselect Security updates notification and click Next

Please note that in Fusion Applications each Applications will include multiple Weblogic Managed Servers, admin servers etc and since our Virtual Machine is not having sufficient capacity to host all these applications, we selected “Oracle Human Capital Management”. You can select any product which you wish to configure first. Click Details to see the topology details for the selected module.

Click Close and then Next on configurations screen.

In this screen provide any name to this provisioning plan. Click Next

Since we entered “weblogic” as Node Manager login earlier, provide its credentials.

Installers Directory Location: The stage or repository location. In our case /mnt/fusion

Oracle Fusion Applications Home: Provide base location for the installation. /app/fusion

Enter /app/fusion/instance for Application Configuration Directory.

As we have noted the location for Webgate Library earlier in Previous post, enter /app/fusion/oam_lib

We had created IDM Properties file /app/fusion/bea_default/Oracle_IAM/idmtools/bin/idmDomainConfig.param in previous post.

RDP Password: oracle123

Click Next

Accept default values. Click Next

Enter database details in this screen. Click Next

Enter same password. Lets’ keep oracle123 again J

This will only accept earlier entered ODI Supervisor password. In our case, nothing to worry since we have all passwords as oracle123

Password: oracle (it validated this password)

We are going to have single host for all domains. Enter our hostname “fusion” and click Next

Deselect DMZ, enter host as fusion, domain as fusion.local. Click Next

Change at least the internal names as above to keep it simple. Click Next

Since we are going to have the simplest installation, deselect Load Balancing. Click Next

Deselect Proxy. Click Next

Entered following values only, remaining were populated from idmDomainConfig.param file

Super User Name: orcladmin

Checked “Create Administrators Group”, “Create Monitors Group”, “Create Operators Group”

Entered oracle123 in both password fields

Unchecked “Identity Store Enabled SSL”, “OIM Endpoint Enabled SSL”

OIM Administrator User Name: xelsysadmin

OIM Administrator Password: oracle123

OIM Managed Server port: 7004

OIM Endpoint Host: fusion (this is because we did not configure load balancing)

OIM Endpoint Port: 7777 (again because we didn’t configure load balancing)

IDM Keystore file: Create a dummy file anywhere on Linux and enter its path here.

[oracle@fusion]$ touch /app/fusion/provisioining/dummy

IDM KeyStore Password: Enter any value

OAM Administrator User Name: oamadmin

OAM Administrator Password: oracle123 (as defined earlier)

OAM AAA Server Host: fusion

OAM AAA Server Port: 5575 (Default and also defined earlier)

Access Server Identifier: wls_oam1

Unchecked OPSS Policy Store SSL Enabled

Deselect Secondary OAM.

OAM Security Mode: Open

Webgate password: oracle123 (as entered earlier in previous post)

Remaining fields already poputed due to idmDomainConfig.param file. Deselect last 2 options. Click Next

Enter Database Details. Also enter DEV_MDS and password (defined earlier during installation in previous post). Click Next

Click Finish to complete creating the plan.

Next: Provisioning an Applications Environment

Installing Oracle Fusion Applications – steps

1. Installing Fusion Applications Provisioning Framework
   
2. Installing Oracle 11g Database (Applications Transactional Database)
   
3. Running Oracle Fusion Applications Repository Creation Utility (Applications RCU)
   
4. Creating another database for Oracle  Identity Management Infrastructure (optional)
   
5. Running Repository Creation Utility (RCU) for Oracle Identity Management components
6. Installing Oracle Identity and Access Management Components
   
7. Configuring Oracle Identity and Access Management components
   
8. Preparing Identity and Policy Stores
   
9. Creating a New Provisioning Plan
   
10. Provisioning an Applications Environment

Oracle Fusion Applications installation: Running Repository Creation Utility (RCU) for Oracle Identity Management Components

Previous: Running Oracle Fusion Applications Repository Creation Utility (Applications RCU)

Important Note: We are NOT creating a separate database to host Oracle Identity Management schemas. Instead we will create these schemas in our Fusion Database (fusiondb) itself since the schema names are distinct compared to Fusion Application schemas. We will save a lot of Memory for our Virtual Machine.

In order to run RCU for Identity Management on same database, we must set the open_cursors parameter to 800.

Open a new database session and set following values.

SQL> show parameter open_cursors

NAME TYPE VALUE

———————————— ———– ——————————

open_cursors integer 500

SQL> alter system set open_cursors=800 scope=both sid=’*';

System altered.

Launch Repository Creation Utility for Oracle Identity Management from <Framework_location>/fmw_rcu/bin location

[oracle@fusion bin]$ /app/fusion/provisioning/fmw_rcu/bin/rcu &

Click Next

Select Create and click Next

Enter the database information (for same fusiondb database as entered earlier. Click Next

Next it will check for required prerequisites. Upon successful check click OK

Click Idnentity Management. It will select few other required components also automatically. Click Next

Again it will check prerequisites for the selected components. Upon successful check click OK

Enter same password oracle123 for ease of remembering. Click Next

No need to change anything in this screen. Click Next

Click OK

It will create the required tablespaces. Once finished click OK

On this summary screen click Create to create required schemas and load data.

This will be quicker than earlier RCU. Once finished proceed to next screen.

Finally it will display Completion Summary. Clock Close

Next: Installing Oracle Identity and Access Management Components

Installing Oracle Fusion Applications – steps

1. Installing Fusion Applications Provisioning Framework
   
2. Installing Oracle 11g Database (Applications Transactional Database)
   
3. Running Oracle Fusion Applications Repository Creation Utility (Applications RCU)
   
4. Creating another database for Oracle  Identity Management Infrastructure (optional)
   
5. Running Repository Creation Utility (RCU) for Oracle Identity Management components
6. Installing Oracle Identity and Access Management Components
   
7. Configuring Oracle Identity and Access Management components
   
8. Preparing Identity and Policy Stores
   
9. Creating a New Provisioning Plan
   
10. Provisioning an Applications Environment

Oracle Fusion Applications Installation: Running Fusion Applications Repository Creation Utility (Apps RCU)

Previous: Installing Oracle 11g Database (Applications Transactional Database)

Create a directory on physical partition and call it APPS_RCU_HOME

[oracle@fusion linux]$ mkdir /app/fusion/provisioning/apps_rcu

Go to repository_location/installers/apps_rcu and locate the rcuHome_fusionapps_dbinstall.zip file. This file was staged when you created the installer repository.

Extract the contents of rcuHome_fusionapps_dbinstall.zip to a directory (APPS_RCU_HOME) on the database server. All dependent components that Applications RCU needs are included in this zipped file.

[oracle@fusion apps_rcu]$ cd /app/fusion/provisioning/apps_rcu

[oracle@fusion apps_rcu]$ unzip /mnt/fusion/installers/apps_rcu/linux/rcuHome_fusionapps_linux.zip

Create a temporary directory on the database server. Make a note of the location.

You will need to enter this location when you specify a value for FUSIONAPPS_DBINSTALL_DP_DIR

[oracle@fusion apps_rcu]$ mkdir /app/fusion/provisioning/apps_rcu/dp_dir

Locate and copy APPS_RCU_HOME/rcu/integration/fusionapps/export_

fusionapps_dbinstall.zip to the directory you specified for FUSIONAPPS_

DBINSTALL_DP_DIR.

Unzip export_fusionapps_dbinstall.zip to FUSIONAPPS_DBINSTALL_DP_DIR.

[oracle@fusion dp_dir]$ cd /app/fusion/provisioning/apps_rcu/dp_dir

[oracle@fusion dp_dir]$ unzip /app/fusion/provisioning/apps_rcu/rcu/integration/fusionapps/export_fusionapps_dbinstall.zip

Go to APPS_RCU_HOME/rcu/integration/biapps/schema and locate the

otbi.dmp file.

Copy otbi.dmp to FUSIONAPPS_DBINSTALL_DP_DIR (where you unzipped the

contents of export_fusionapps_dbinstall.zip).

[oracle@fusion dp_dir]$ cp -p ../rcu/integration/biapps/schema/otbi.dmp /app/fusion/provisioning/apps_rcu/dp_dir/

Launch Repository Creation Utility (RCU)

[oracle@fusion bin]$ cd /app/fusion/provisioning/apps_rcu/bin

[oracle@fusion bin]$ ./rcu

Click Next

Click Next

Enter same details as entered while creating the database in previous step. Click Next

This screen will check pre-requisites. Click Ok once successful.

Select all components in this Window.

It will look as above when you collapse all parent values. Click Next

Important Note: If you had reduced SGA and PGA size after DB installation then this prerequisites check may fail with following error.

RCU-6083:Failed – Check prerequisites requirement for selected component:FUSIONAPPS

RCU-6107:DB Init Param Prerequisite failure for: pga_aggregate_target

Current Value is 2147483648. It should be greater than or equal to 4294967296.

RCU-6107:DB Init Param Prerequisite failure for: sga_target

Current Value is 2147483648. It should be greater than or equal to 9663676416.

 

To fix this issue, we need to modify the minimum requirement of SGA and PGA in installer pre-requisite config file located at /app/fusion/provisioning/apps_rcu/rcu/integration/fusionapps/fusionapps.xml

 

Change these values as follows.

<DBPrerequisite COMPARE_OPERATOR=”GE” DATA_TYPE=”NUMBER” PREREQ_TYPE=”InitParameter”>

<ValidIf DBTYPE=”ORACLE”/>

<PrereqIdentifier>sga_target</PrereqIdentifier>

<PrereqValue>2147483648</PrereqValue>

</DBPrerequisite>

…

<DBPrerequisite COMPARE_OPERATOR=”GE” DATA_TYPE=”NUMBER” PREREQ_TYPE=”InitParameter”>

<ValidIf DBTYPE=”ORACLE”/>

<PrereqIdentifier>pga_aggregate_target</PrereqIdentifier>

<PrereqValue>2147483648</PrereqValue>

</DBPrerequisite>

The prerequisites check should finish successfully. Click Ok

Enter same password for keeping it simplefor now. Click Next

Open another terminal window. Create following directories for custom environment variables.

[oracle@fusion database]$ mkdir /app/fusion/database/applcp

[oracle@fusion database]$ mkdir /app/fusion/database/appllog

[oracle@fusion database]$ mkdir /app/fusion/database/keyflexcombfilter

[oracle@fusion database]$ mkdir /app/fusion/database/obieebkp

Specify these following values in the same screen under Fusion Applications Component (expect first value which is the temporary DP directory created before /app/fusion/provisioning/apps_rcu/dp_dir)

Supervisor Password: You must enter the same password you set up as ODI SUPERVISOR in Applications RCU. Since we kept all passwords as oracle123, nothing much to remember. Enter the same password.

Work Repository Password: Default = None. You must enter the same password set up as ODI SUPERVISOR in Applications RCU.

Oracle Transactional BI

Directory on the database server where Oracle Transactional Business Intelligence import and export files are stored. Enter /app/fusion/provisioning/apps_rcu/dp_dir again.

Change nothing in this page. Click Next

Click OK

Now it will create the required tablespaces. Click OK

Once Tablespaces are created, next Summary screen will appear for creating the required Schemas. Click Create

It may take a couple of hours or even more based on the available memory and CPU for the VM and host machine.

Once finished, a completion summary screen will appear. It will show details for each components and completion states. Click Close

Next: Running Repository Creation Utility (RCU) for Oracle Identity Management components

Installing Oracle Fusion Applications – steps

1. Installing Fusion Applications Provisioning Framework
   
2. Installing Oracle 11g Database (Applications Transactional Database)
   
3. Running Oracle Fusion Applications Repository Creation Utility (Applications RCU)
   
4. Creating another database for Oracle  Identity Management Infrastructure (optional)
   
5. Running Repository Creation Utility (RCU) for Oracle Identity Management components
6. Installing Oracle Identity and Access Management Components
   
7. Configuring Oracle Identity and Access Management components
   
8. Preparing Identity and Policy Stores
   
9. Creating a New Provisioning Plan
   
10. Provisioning an Applications Environment

Oracle Fusion Applications Installation: Installing Transactional Database

Previous: Installing Fusion Applications Provisioning Framework

To install Applications Transactional Database we need to run Oracle Fusion Applications Provisioning Wizard from <framework_location>/provisioning/bin

<framework_location> is same what we mentioned in previous post. i.e. /app/fusion

[oracle@fusion $ cd /app/fusion/provisioning/bin

We need to temporarily set JAVA_HOME to jdk6 directory shipped with the installation media

[oracle@fusion bin]$ export JAVA_HOME=/mnt/fusion/jdk6

[oracle@fusion bin]$ ./provisioningWizard.sh

Click Next

Click Next

Deselect security updates notification. Click Next

Provide database listener port (default is 1521, if you change this port please note to enter the new port in future screens where we have mentioned 1521)

Installer directory location will be the same as you created the stage setup or provisioning repository.

Provide a location for Oracle Base. Press TAB, next values should automatically be populated, if not, go back and click next again.

Enter dba as OSDBA group.

We have specified “fusiondb” as our database name. please note that if you are using any other name then don’t forget to change it in further screens whenever we have mentioned fusiondb.

You can keep simple password like “oracle” also but try to keep something like “oracle123” for all passwords so that it will follow requirements for all further passwords and it will be easier for you to remember all passwords in further installations.

Next it will finish the pre-requisite checks. Click Next

On next page, you can Save the summary. Click Install to start the installation.

Note: This will install the software as well as create a database named “fusiondb” which we provided earlier.

At this point it will prompt for running root.sh as root user.

Do not press Ok until run the following as root user in separate terminal window. Once following is executed, press Ok to continue.

[root@fusion ~]# /app/fusion/database/product/11.2.0/dbhome_1/root.sh

Check /app/fusion/database/product/11.2.0/dbhome_1/install/root_fusion_2012-01-1 5_12-29-28.log for the output of root script

[root@fusion ~]# tail -f /app/fusion/database/product/11.2.0/dbhome_1/install/root_fusion_2012-01-15_12-29-28.log

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /app/fusion/database/product/11.2.0/dbhome_1

 

Creating /etc/oratab file…

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Finished product-specific root actions.

It may take a few hours (at least on a non-high end host machine) and then installation will finish.

Important Note: After installation is finished we reduced the SGA and PGA (from 10GB to 2GB). We are doing this since we are running on a VM and have allocated total 2.5 or 3 GB RAM only. This is not recommended for production installation but since this is only a demo/development installation, you can safely do this. If you wish to keep it 10GB you can do it but this will increase the swap usage exponentially.

Since we are reducing the size of SGA and PGA, next step (RCU) may fail in pre-requisite check. So we will need to change the pre-req check xml file to look for lower value. We will explain this in next post.

Next:  Running Oracle Fusion Applications Repository Creation Utility (Applications RCU)

Installing Oracle Fusion Applications – steps

1. Installing Fusion Applications Provisioning Framework
   
2. Installing Oracle 11g Database (Applications Transactional Database)
   
3. Running Oracle Fusion Applications Repository Creation Utility (Applications RCU)
   
4. Creating another database for Oracle  Identity Management Infrastructure (optional)
   
5. Running Repository Creation Utility (RCU) for Oracle Identity Management components
6. Installing Oracle Identity and Access Management Components
   
7. Configuring Oracle Identity and Access Management components
   
8. Preparing Identity and Policy Stores
   
9. Creating a New Provisioning Plan
   
10. Provisioning an Applications Environment

Nowadays DBAs face requirements for migrating databases across different platforms like from Windows to Linux or Solaris or vice versa. While doing this we would require to use transportable tablespaces or cross-platform transportable tablespaces. In order to decide whether the migration would be simple or will involve file conversion, following command is used to identify platform endian format. If the endian formats are same then the process pretty straightforward.

SQL> select * from v$transportable_platform order by platform_id;

PLATFORM_ID	PLATFORM_NAME	ENDIAN_FORMAT
1	Solaris[tm] OE (32-bit)	Big
2	Solaris[tm] OE (64-bit)	Big
3	HP-UX (64-bit)	Big
4	HP-UX IA (64-bit)	Big
5	HP Tru64 UNIX	Little
6	AIX-Based Systems (64-bit)	Big
7	Microsoft Windows IA (32-bit)	Little
8	Microsoft Windows IA (64-bit)	Little
9	IBM zSeries Based Linux	Big
10	Linux IA (32-bit)	Little
11	Linux IA (64-bit)	Little
12	Microsoft Windows 64-bit for AMD	Little
13	Linux 64-bit for AMD	Little
14	HP Open VMS	Little
15	Apple Mac OS	Big

Oracle SQL Tuning Tips

consideration when writing an SQL statement is that it returns a correct result. The second is that it be the most efficient for a given situation.  You can use many different SQL statements to achieve the same result. It is often the case that only one statement will be the most efficient choice in a given situation.

Remember that processing SQL is a sequence of Parse (syntax check and object resolution), Execution (required reads and writes), and Fetch (row results retrieved, listed, sorted, and returned). SQL “tuning” consists, quite simply, of reducing one or more of them.

Note: generally Parse is the greatest time and resource hog. Parse overhead can be minimized by the use of Procedures, Functions, Packages, Views, etc.

Inadequate performance can have a significant cost impact on your business. A poor performing system and application can result in customer dissatisfaction, reduced productivity, and high costs. It is absolutely critical that the system’s performance is operating at its peak levels.

Following are some general tips that often increase SQL statement efficiency. Being general they may not apply to a particular scenario.

• Tuning SQL should only be done after your code is working correctly. Be aware that there is an inevitable tug-of-war between writing efficient SQL and understandable SQL.
• Ensure repeated SQL statements are writtenabsolutely identicallyto facilitate efficient reuse: re-parsing can often be avoided for each subsequent use of an identical statement.
  Writing best practices: all SQL verbs in upper-case i.e. SELECT; separate all words with a single space; all SQL verbs begin on a new line; SQL verbs aligned right or left within the initial verb; set and maintain a table alias standard; use table aliases and when a query involves more than one table prefix all column names with their aliases. Whatever you do, be consistent.
• o Use bind variables: The values of bind variables do not need to be the same for two statements to be considered identical. Bind variables are not substituted until a statement has been successfully parsed.
• o Use standard approach to table aliases. If two identical SQL statements vary because an identical table has two different aliases, then the SQL is different and will not be re-use or shared.
• o Use table aliases and prefix all column names by their aliases when more than one table is involved in a query. This reduces parse time and prevents future syntax errors if someone adds a column to one of the tables with same name as a column in another table. (ORA-00918: COLUMN AMBIGUOUSLY DEFINED)
• Code the query as simply as possible i.e. no unnecessary columns are selected, no unnecessary GROUP BY or ORDER BY.
  It is the same or faster to SELECT by actual column name(s). The larger the table the more likely the savings.
  Use:
  SELECT customer_id, last_name, first_name, street, city FROM customer;
  Rather than,
  SELECT * FROM customer;
• Beware of WHERE clauses which do not use indexes at all. Even if there is an index over a column that is referenced by a WHERE clause included in this section, Oracle will ignore the index. All these WHERE clause can be re-written to use an index while returning the same values. In other words, Do not perform operations on database objects referenced in the WHERE clause:
  Use:
  SELECT client, date, amount FROM sales WHERE amount > 0;
  Rather than:
  SELECT client, date, amount FROM sales WHERE amount != 0;
  ———————————————
  Use:
  SELECT account_name, trans_date, amount FROM transaction WHERE account_name LIKE ‘CAPITAL%’;
  Rather than:
  SELECT account_name, trans_date, amount FROM transaction WHERE SUBSTR(account_name,1,7)=‘CAPITAL’;
  ———————————————
  Use:
  SELECT account_name, trans_date, amount FROM transaction WHERE amount > 0;
  Rather than:
  SELECT account_name, trans_date, amount FROM transaction WHERE amount NOT=0;
  ———————————————
  Use:
  SELECT account_name, trans_date, amount FROM transaction WHERE amount < 2000;
  Rather than:
  SELECT account_name, trans_date, amount FROM transaction WHERE amount + 3000 < 5000;
  ———————————————
  Use:
  SELECT account_name, trans_date, amount FROM transaction WHERE account_name = ‘AMEX’ AND account_type = ‘A’;
  Rather than:
  SELECT account_name, trans_date, amount FROM transaction WHERE account_name|| account_type=‘AMEXA’;
• Don’t forget to tune views. Views are SELECT statements and can be tuned in just the same way as any other type of SLECT statement can be. All tuning applicable to any SQL statement are equally applicable to views.
• Avoid including a HAVING clause in SELECT statements. The HAVING clause filters selected rows only after all rows have been fetched. Using a WHERE clause helps reduce overheads in sorting, summing, etc. HAVING clauses should only be used when columns with summary operations applied to them are restricted by the clause.
  Use:
  SELECT city FROM country WHERE city!= ‘Vancouver’ AND city!= ‘Toronto’; GROUP BY city;
  Rather than:
  SELECT city FROM country GROUP BY city HAVING city!= ‘Vancouver’ AND city!= ‘Toronto’;
• Minimize the number of table lookups (subquery blocks) in queries, particularly if our statements include subquery SELECTs or multicolumn UPDATEs.
  Use:
  SELECT emp_name FROM emp WHERE (emp_cat, sal_range) = (SELECT MAX(category), MAX(sal_range) FROM emp_categories) AND emp_dept = 0020;
  Rather than:
  SELECT emp_name FROM emp WHERE emp_cat=(SELECT MAX(category) FROM emp_categories) AND emp_range = (SELECT MAX(sal_range) FROM emp_categories) AND emp_dept = 0020;
• When writing a sub-query (a SELECT statement within the WHERE or HAVING clause of another SQL statement):
• o Use a correlated (refers to at least one value from the outer query) sub-query when the return is relatively small and/or other criteria are efficient i.e. if the tables within the sub-query have efficient indexes.
• o Use a non-correlated (does not refer to the outer query) sub-query when dealing with large tables from which you expect a large return (many rows) and/or if the tables within the sub-query do not have efficient indexes.
• o Ensure that multiple sub-queries are in the most efficient order.
• o Remember that rewriting a sub-query as a join can sometimes increase efficiency.
• When doing multiple table joins consider the benefits/costs for each of EXISTS, IN, and table joins. Depending on your data one or another may be faster.
  Note: IN is usually the slowest.
  Note: When most of the filter criteria are in the sub-query IN may be more efficient; when most of the filter criteria are in the parent-query EXISTS may be more efficient.
  The following queries return the employee names from each department in department category ‘A’:
  SELECT emp_name FROM emp E WHERE EXISTS (SELECT ‘X’ FROM dept WHERE dept_no = E.dept_no AND dept_cat = ‘A’);
  ———————————————
  SELECT emp_name FROM emp E WHERE dept_no IN (SELECT dept_no FROM dept WHERE dept_no = E.dept_no AND dept_Cat = ‘A’);
  ———————————————
  SELECT emp_name FROM dept D, emp E WHERE E.dept_no = D.dept_no AND D.dept_cat = ‘A’;
• Where possible use EXISTS rather than DISTINCT, to avoid full table scan as DISTINCT operator causes Oracle to fetch all rows satisfying the table join and then sort and filter out duplicate values. EXISTS is a faster alternative, because the Oracle optimizer realizes when the subquery has been satisfied once, there is no need to proceed further and the next matching row can be fetched.
  Use:
  SELECT S.id, S.description FROM small_table S WHERE EXISTS (SELECT NULL FROM big_table B WHERE B.id = S.id);
  Rather than:
  SELECT DISTINCT S.id, S.description FROM small_table S, big_table B
  WHERE S.id = B.id;
• Where possible use a non-column expression (putting the column on one side of the operator and all the other values on the other). Non-column expressions are often processed earlier thereby speeding the query.
  Use:
  WHERE SALES < 1000/(1 + n);
  Rather than:
  WHERE SALES + (n * SALES) < 1000;
• Where possible use UNION ALL rather than using UNION. The UNION clause forces all rows retuned by each portion of the UNION to be sorted and merged and duplicates to be filtered out before the first row is returned. A UNION ALL simply returns all rows including duplicates and does not have to perform any sort, merge or filter. If your tables are mutually exclusive(include no duplicate records), or you don’t care if duplicates are returned, the UNION ALL is much more efficient.
  USE:
  SELECT acct_num, balance_amt FROM debit_transactions WHERE tran_date = ’31-DEC-95′
  UNION ALL
  SELECT acct_num, balance_amt FROM credit_transactions WHERE tran_date = ’31-DE -95′;
  Rather than:
  SELECT acct_num, balance_amt FROM debit_transactions WHERE tran_date = ’31-DEC-95′
  UNION
  SELECT acct_num, balance_amt FROM credit_transactions WHERE tran_date = ’31-DE -95′;
• Consider using DECODE to avoid having to scan the same rows repetitively or join the same table repetitively. Note, DECODE is not necessarily faster as it depends on your data and the complexity of the resulting query. Also, using DECODE requires you t change your code when new values are allowed in the field.
  USE:
  SELECT COUNT(DECODE(status,’Y',’X',NULL)) Y_count,
  COUNT(DECODE(status,’N',’X',NULL)) N_count
  FROM emp WHERE emp_name LIKE ‘SMITH%’;
  Raher than:
  SELECT COUNT(*) FROM emp WHERE status = ‘Y’ AND emp_name LIKE ‘SMITH%’;
  ———————
  SELECT COUNT(*) FROM emp WHERE status = ‘N’ AND emp_name LIKE ‘SMITH%’;
• Oracle automatically performs simple column type conversions(or casting) when it compares columns of different types. Depending on the type of conversion, indexes may not be used. Make sure you declare your program variables as the same type as your Oracle columns, if the type is supported in the programming language you are using.
  Use:
  SELECT emp_no, emp_name, sal FROM emp WHERE emp_no = ’123′;
  HERE if emp_no indexed numeric, then after implicit conversion query will be:
  SELECT emp_no, emp_name, sal FROM emp WHERE emp_no = TO_NUMBER(’123′);
  Thus, index is used in this case.
  Don’t use:
  SELECT emp_no, emp_name, sal FROM emp WHERE emp_type = 123;
  HERE if emp_type is indexed varchar2, then after implicit conversion query will be:
  SELECT emp_no, emp_name, sal FROM emp WHERE TO_NUMBER(emp_type) = 123;
  Thus, index will not be used in this case.
• The most efficient method for storing large binary objects, i.e. multimedia objects, is to place them in the file system and place a pointer in the DB.
• B-Tree Indexes do not store entries for NULL, so IS NULL is not indexable, but IS NOT NULL is indexable and thus if a huge table contains very few not null values then you should go for B-Tree indexes. On the other hand bitmap indexes support IS NULL condition.
  SELECT * FROM big WHERE status IS NOT NULL; (Use B-tree index in this case)
• Avoid using functions on indexed columns unless a function-based index is created; as it leads to full table scan even though index exists on the column.

Avoid using the following:

• Boolean operators >, <, >=, <=, IS NULL, IS NOT NULL
• NOT IN, !=
• Like ‘%pattern’, not exists
• Calculations on unindexed columns or (use union instead)
• Having (use a WHERE clause instead when appropriate)

Do use the following:

• Enable aliases to prefix all columns
• Place indexed columns higher in the WHERE clause
• Use SQL Joins instead of using sub-queries
• Make the table with the least number of rows, the driving table, by making it first in the FROM clause

Other important points for SQL Tuning

• Establish a tuning environment that reflects your production database
• Establish performance expectations before you begin
• Always Design and develop with performance in mind
• Create Indexes to support selective WHERE clauses and join conditions
• Use concatenated indexes where appropriate
• Consider indexing more than you think you should, to avoid table lookups
• Pick the best join method
• Nested loops joins are best for indexed joins of subsets
• Hash joins are usually the best choice for “big” joins
• Pick the best join order
• Pick the best “driving” table
• Eliminate rows as early as possible in the join order
• Use bind variables. Bind variables are key to application scalability
• Use Oracle hints where appropriate
• Compare performance between alternative syntax for your SQL statement
• Consider utilizing PL/SQL to overcome difficult SQL tuning issues
• Consider using third party tools to make the job of SQL tuning easier

Performing these steps is easy and provides a tremendous benefit and performance boost. Follow these simple steps and you can increase your system performance.

Using SQL*Plus

SQL*Plus is a command-line tool that provides access to the Oracle RDBMS.

SQL*Plus enables you to:

• Enter SQL*Plus commands to configure the SQL*Plus environment
• Startup and shutdown an Oracle database
• Connect to an Oracle database
• Enter and execute SQL commands and PL/SQL blocks
• Format and print query results

SQL*Plus is available on several platforms. In addition, it has a web-based user

interface, iSQL*Plus.

SQL*Plus is a client terminal software allowing users to interact with Oracle server to manipulate data and data structures. Users type in SQL statements in SQL*Plus that send statements to Oracle server. Oracle server then validates and executes the statements on its databases. The query results are returned to SQL*Plus and displayed to the user.  Besides sending SQL statements to the server, SQL*Plus also saves them into a local buffer and allow users to view and change the statements. The following figure illustrates the process.

After you login into SQL*Plus, at the SQL prompt, you can begin typing any SQL command. Upon hitting return (i.e., enter key) the SQL prompt will change to line number prompts. When you are finished typing a command, type / or RUN to execute the SQL command. Also, a semicolon at the end of the SQL command will execute the command immediately after hitting return. In addition to SQL commands, /, and RUN, you can also executes SQL*Plus file commands.

SQL*PLUS Commands Quick Reference

Below table shows, SQL*Plus commands available in the command-line interface. Not all commands or command parameters are shown.

SQL*Plus file command allow you to execute commands (or programs) stored in an external file, input or output data from/to a file, and save SQL commands typed during current session.

Some SQL*Plus file commands are:

• SAVE filename. This allows you to save buffer contents into a file.
• START filename. This allows you to execute a batch of SQL statements stored in a file.
• SPOOL filename. This allows you save SQL statements together with their outputs to a file.
• GET filename. This retrieve a file and places it into the buffer.
• @ filename. This allows you to execute a PL/SQL procedure(s) stored in a file.

Recall that the previously executed commands (in current SQL*Plus session) are stored in the local buffer. One way to change an SQL statement in the buffer is by using the line editor. The following are a list of line edit commands.

• LIST or L–Lists the contents of the buffer
• LIST n or L n–Lists the contents of line number n in the buffer and makes the line current
• LIST * or L *–Lists the current line
• LIST m n–Lists the range from m to n line
• Append text or A text–Adds to the end of the current line (e.g., “A ,” adds a comma to the end of line
• INPUT or I–Adds one or more lines after the current line so you can begin adding the text.
• CHANGE /text–Deletes text from the current line
• CHANGE /oldtext/newtext–Replaces oldtext with newtext in the current line
• DEL — Deletes the current line

Besides line editor, you can also use the vi editor if you are a fan of Unix editor!.

To invoke the vi editor, type Edit at the SQL Prompt.   Multiple SQL commands can be typed in vi editor. End each SQL command (except the last one) with a semicolon. After exiting notepad, type Start to run all of the commands.

Run SQL statements in a batch

To run SQL commands in a batch, you can put all your SQL commands into a text file and execute these commands in this file in SQL*PLUS.

• Use your favorite editor to type in your SQL queries into a text file.

For Example,

$ more table.sql
DROP TABLE employee
/

commit
/

CREATE TABLE employee (
empno INTEGER NOT NULL,
name VARCHAR2(50) NOT NULL,
sal REAL NOT NULL,
primary key (empno));
/

INSERT INTO employee VALUES (1, ‘Jack’, 6000);
INSERT INTO employee VALUES (2, ‘Tom’,  6000);
INSERT INTO employee VALUES (3, ‘John’, 6000);
INSERT INTO employee VALUES (4, ‘Jane’, 6000);
/

UPDATE employee SET sal=500 WHERE name=’Jack’
/

CREATE INDEX test_index on employee(sal)
/
$

• Connect into SQL*Plus, and run the batch of commands. For example, assume that you name the SQL file as table.sql.

SQL> START table.sql;

Output results

• You can record your SQL command outputs to a file for output or editing purpose.

SQL> SPOOL <your file name>

For example,

SQL> SPOOL myoutput.out

All SQL commands and their outputs after this command are written into the file myoutput.out that by default is stored in the current working directory where you invoked SQL*Plus.

• To end recording, use the following command:

SQL> SPOOL OFF

DUAL and select the current time

DUAL is the dummy table, mostly used to view the results from functions and calculations. The built-in function SYSDATE returns a DATE value containing the current date and time on your system. (Note Oracle is a client-server architecture and SQL*Plus is the client. SYSDATE gives you the time of the Unix system which you telnet in. It may NOT be the time of Oracle server unless you telnet into the machine running Oracle server.)

For example,

SQL> SELECT TO_CHAR(SYSDATE , 'Dy DD-Mon-YYYY HH24:MI:SS') as "Current Date/Time" FROM DUAL;;

Result:

Current Time

------------------------------------------------------------------------

Mon 15-July-2002 10:01:29

• TO_CHAR is a function to format a value.
• DUAL is built-in relation in Oracle which serves as a dummy relation to put in the FROM clause when nothing else is appropriate. For example, try “SELECT 2+2 FROM DUAL;”
• To format a number attribute to a dollar format, use the column <attribute> format <format>:

       SQL> COLUMN salary FORMAT $999,999

• To indicate the displayed width of a character string attribute, use the column <attribute> format <A’format>. For example, set the width of the name attribute to 8 characters.

       SQL> COLUMN name FORMAT A8

If a name is longer than 8 characters, the remaining is displayed at the second line (or several lines)

• The set command can be used to change the default number of lines per page (14) and the number of characters per line (80).

For example, to set the number of lines per page to 60, use the following command:

   SQL> SET PAGESIZE 60

• All formatting remain active until they are cleared or reset or after you exit from SQL*Plus.

   SQL> CLEAR COLUMN

• If you forget a specific SQL command you could enter

SQL> HELP <the SQL command>;

You could also find out all commands by entering:

SQL> HELP menu;

• Sometimes when you get something fuzzy, you  can try the following

SQL> SET SERVEROUTPUT ON
SQL> SET ARRAYSIZE 1

Oracle Data Export and Import utility

The Import and Export utilities work together; Export sends database definitions and actual data to an export file and Import can read the file to perform many different tasks. You can use Export and Import for many important database tasks, such as restoring a table, generating CREATE scripts, copying data among Oracle databases, migrating among Oracle versions, and moving tables from one schema to another.

Oracle’s export (exp) and import (imp) utilities are used to perform logical database backup and recovery. When exporting, database objects are dumped to a binary file which can then be imported into another Oracle database.

These utilities can be used to move data between different machines, databases or schema. However, as they use a proprietary binary file format, they can only be used between Oracle databases. One cannot export data and expect to import it into a non-Oracle database.

Various parameters are available to control what objects are exported or imported. To get a list of available parameters, run the exp or imp utilities with the help=yes parameter i.e. “exp help=yes” or “imp help=yes“.

The export/import utilities are commonly used to perform the following tasks:

• Back up and recover objects. Perhaps the most widely used Export and Import feature is the capability to serve as a useful backup solution. Most Oracle installations use full online or hot backups to back up the entire database in the event of a CPU, network, or disk failure. This works well because all data can be restored to the very millisecond that the database crashed. However, Export and Import provide for extra safety. If just one table in the entire database needs to be recovered during a full backup, the entire database would have to be recovered on another machine, and the one table would be copied back over into the original database. This incredibly time- and resource-consuming task could become a real headache in distributed databases. Export and Import can save you this trouble by exporting the entire database and importing just the tables that need to be recovered. Now that point-in-time recovery is an option for tablespaces in Oracle8, Export and Import can also serve as a backup and recovery mechanism for the entire database.
• Copy objects among schemas. You can use Export and Import to copy all object types such as tables, indexes, grants, procedures, and views from one schema to another. You can specify which objects you want to move, and provide the FROM and TO schemas. Oracle does the rest automatically.
• Copy objects among Oracle databases. You can export a table in one Oracle database and import that table into another. Many businesses export key tables and send the export files to a remote office location, which can then use Import to load the data into its local database. This is a form of one-way data replication to remote locations.
• Generate CREATE scripts. Export and Import contain powerful options that allow you to generate CREATE scripts for tables, partitions, indexes, object-type definitions, views, grants, constraints, rollback segments, tablespaces, and all other objects in the database. This safeguards the structure of your objects, in case one gets corrupted or deleted.
• Migrate databases from one Oracle version to another. You can upgrade (or downgrade) the version of Oracle by using the Export and Import utilities. For example, you can export an Oracle7 database, copy the export file to an Oracle8 database server, and then import into the Oracle8 server. This process, called migration, makes the data and application function in an Oracle8 environment. Of course, we always recommend complete testing before relying on the results of a migration.
• Defragment a tablespace. Fragmentation occurs when tables and indexes are created, dropped, enlarged, and reduced in size over time. Fragmentation also occurs when an object’s storage parameters are sized improperly. To defragment a tablespace, you can export a tablespace, coalesce the free space (or drop and recreate the tablespace), and import the objects again.

Because of the special binary format, files which had been created by the EXPORT utility can only be read by IMPORT. Both tools are only used to maintain ORACLE database objects.

SQL*Loader is similar to Import in that it can load data; the main difference is that Import can read only Oracle export files. SQL*Loader can read text files generated by non-Oracle databases.To load data from other systems into the database you have to use SQL*LOADER or other programs. To store data from database into readable operating system files you may use e.g. REPORTS, SQLPLUS.

While still supported on 10g the original import (imp) and export (exp) utilities were replaced in 10g with the faster and more advanced Data Pump-based import-export, impdp and expdp.

NOTE: It is generally advised not to use exports as the only means of backing-up a database. Physical backup methods (for example, when you use RMAN) are normally much quicker and supports point in time based recovery (apply archivelogs after recovering a database). Also, exp/imp is not practical for large database environments.

On 11g just imp will be still available, so if you using 10g, now is the time to starting using data pump.

Prerequisites/Requirements

exp and imp are utilities present in the $ORACLE_HOME/bin directory and are installed when Oracle is installed. Their prime purpose is to move logical objects out of and into the database respectively – for example dumping all of the tables owned by a user to a single file is achieved using the exp utility. It is important to distinguish between dumping data in this manner and backing up the database which is normally achieved using the rman utility.

$ORACLE_HOME, $ORACLE_SID and $PATH environment variables need to be set appropriately in a unix environment and the %ORACLE_SID% environment variable needs to be set in a windows environment, before executing these commands. If you login as the owner of the oracle software (usually the oracle user on unix) these are likely to have been set in the users profile. The Oracle Database instance needs to be up in order to export/import data. Type ‘imp help=y’ or ‘exp help=y’ for a detailed explanation of the available options for these utilities.

It is a prerequisite that oraenv or coraenv (if you are working in a c-shell) was executed before you export-import data. Consider, if you stored data in the database by using an national language support (NLS) other than AMERICAN you have to change your environment before import-export.
Example for a german environment:
NLS_LANG=GERMAN_GERMANY.WE8ISO8859P1
export NLS_LANG

In order to use exp and imp, the catexp.sql script must be run. catexp.sql basically creates the exp_full_database and imp_full_database roles. It is found under $ORACLE_HOME/rdbms/admin:

SQL> @?/rdbms/admin/catexp

catexp is called by catalog.sql. Note: SQL*Plus provides a shortcut to refer to the ORACLE_HOME directory: the question mark (?).

One must have the create session privilege for being able to use exp. If objects of another user’s schema need to be exported, the EXP_FULL_DATABASE role is required.

Before one imports rows into already populated tables, one needs to truncate or drop these tables to get rid of the old data. If not, the new data will be appended to the existing tables or it’ll get rejected. Either way, use the same syntax when you are ready to import the data. One must always DROP existing Sequences before re-importing. If the sequences are not dropped, they will generate numbers inconsistent with the rest of the database.

Note: It is also advisable to drop indexes before importing to speed up the import process. Indexes can easily be recreated after the data was successfully imported.

Control and Configure Export-Import
FULL, OWNER, and TABLE are the three types of exports. A FULL export exports all objects, structures, and data within the database for all schemas. OWNER exports only those objects owned by the specified user accounts. TABLE exports only the specified tables and partitions for specific user accounts.

Because dozens of object types can be exported, it’s important to distinguish what gets exported with each export category:

• With TABLE, the specified table (or cluster or partition) is exported with its indexes, referential integrity constraints, synonyms, and triggers.
• With OWNER specified, all the specified schema’s objects are exported. This includes database links, sequences, packages/procedures/functions, object-type definitions, clusters/tables/partitions, postable actions, synonyms, views, triggers, snapshots/snapshot logs, job queues, and refresh groups.
• With the FULL option, all objects described in OWNER are exported for every user account, in addition to tablespace definitions, profiles, user definitions (and encrypted passwords), roles, resource costs, rollback segment definitions, directory aliases, user history table, and auditing options.

You can use the Export and Import utilities in interactive or non-interactive mode. In interactive mode, Oracle steps you through the process, prompting for basic Import/Export session information. This method, however, limits you to the simple prompts that Oracle provides. You have more flexibility in non-interactive mode because you can supply up to 23 parameters for the Export utility and 24 parameters for the Import utility. The parameters may be supplied at the command line or by specifying a command file that contains all parameters and values. You can automate Export and Import sessions by using a command file, or repeat them over and over again manually.

The syntax of running Export and Import is as follows:
EXP KEYWORD=value or keyword=(value1,value2,…,valueN)
IMP KEYWORD=value or keyword=(value1,value2,…,valueN)
Replace KEYWORD with the parameter, and value with what should be the parameter’s value. For example, use the following to export the entire database:
EXP USERID=SYSTEM/MANAGER FULL=Y

Use the following to import the EMP and DEPT tables owned by the SCOTT user account: IMP USERID=SYSTEM/MANAGER FROMUSER=SCOTT TABLES=(EMP,DEPT)The following examples demonstrate how the imp/exp utilities can be used:
exp scott/tiger file=emp.dmp log=emp.log tables=emp rows=yes indexes=no
exp scott/tiger file=emp.dmp tables=(emp,dept)
imp scott/tiger file=emp.dmp full=yes
imp scott/tiger file=emp.dmp fromuser=scott touser=scott tables=dept

Using a parameter file:
exp userid=scott/tiger@orcl parfile=export.txt
… where export.txt contains:
BUFFER=100000
FILE=account.dmp
FULL=n
OWNER=scott
GRANTS=y
COMPRESS=y
NOTE: If you do not like command line utilities, you can import and export data with the “Schema Manager” GUI that ships with Oracle Enterprise Manager (OEM).
———————————————————————————–
The EXPORT utility may be used in three ways:

• Interactive dialogue
  Simply type exp. You will be prompted for your ORACLE userid, password. All other prompts answer by pressing the return key. This is the easiest way to export all data you stored in the ORACLE database. You may assign other values than the defaults to the parameters but in most cases it is unnecessary.
• Controlled through bypassed parameters
  You may bypass parameters when you export data from the database. Type:
  exp <userid/password> <parameter>=<value>,<parameter>=<value>,…
  Example of exporting scott’s tables EMP and DEPT to file empdept.expdat and storing all messages to file empdept.log:
  exp scott/tiger file=empdept.expdat tables=(EMP,DEPT) log=empdept.log
• Parameterfile controlled
  Instead of typing the parameters on the commandline you may use a parameter file where the parameters are stored. The syntax for this method is: exp <userid/password> parfile=<filename>

Do not ignore warnings and messages. Look at your logfile and repeat EXPORT until you get messages like:

About to export specified tables ...
. exporting table                            EMP       14 rows exported
. exporting table                           DEPT        4 rows exported
Export terminated successfully without warnings.
--------------------------------------------------------------------------------------
Like Export the Import utility is controlled by parameters. To get familiar with these parameters type: imp help=y
You will get a short description of usage and default settings of parameters.
To start IMPORT simply type imp. You will be prompted for your ORACLE userid, password. The next prompts depend on what you answer. In most cases you may answer the prompts by pressing the return key. But the following prompts you have to answer carefully.
Import file: expdat.dmp >
If your data was exported to file expdat.dmp press return, otherwise enter the filename where the exported data resides.
Ignore create error due to object existence (yes/no): yes >
This is a flag to indicate how object creation errors should be handeld. If you import into an existing table and you set IGNORE=Y, rows could be duplicated if they where already present in the table.
Import entire export file (yes/no): yes > no
Username:
If your exportfile consists of more objects than you want to import, enter no. In this case you will be prompted for the Username (this is normally your ORACLE account). Enter table names. Null list means all tables for user
Enter table name or . if done:
After entering the username you will be prompted for table names until you press the return key without entering a table name. Than IMPORT will be started.

Instead of the dialogue method you may use parameters. This is analogous to the methods described for EXPORT.
Examples:
imp <userid/password> tables=(table1,table2)
Tables table1, table2 will be imported from the default file export.dmp into the database.
imp <userid/password> parfile=<filename>
Import will be controlled by the named parameterfile.
After importing you should get messages like:

importing SCOTT's objects into SCOTT
. importing table "DEPT"                                     4 rows imported
. importing table "EMP"                                     14 rows imported
Import terminated successfully without warnings.
-----------------------------------------------------------------------------------

Parameters for the Export utility

Table below lists all parameters that can be used with the Export utility, along with their default values (if any).

Parameters for the Import utility
You can use 24 parameters during an import session. They may be specified in the command line or any specified parameter file. Table below lists all the import parameters.

————————————————————————————–Using imp/exp across different Oracle version

If exp and imp are used to export data from an Oracle database with a different version than the database in which is imported, then the following rules apply:

• 1. exp must be of the lower version
• 2. imp must match the target version

——————————————————————————————–Common Export/Import Errors

• ORA-00001: Unique constraint (…) violated
  You are importing duplicate rows. Use IGNORE=YES to skip tables that already exist (imp will give an error if the object is re-created) OR Perform the following actions:
• 1. Back up the target table.
• 2. Try deleting the target table.
• 3. Run the import for this table again in overwrite mode.
• ORA-01555: Snapshot too old
  Ask your users to STOP working while you are exporting or try using parameter CONSISTENT=NO
• ORA-01562: Failed to extend rollback segment
  Create bigger rollback segments or set parameter COMMIT=Y while importing
• IMP-00015: Statement failed … object already exists…
  Use the IGNORE=Y import parameter to ignore these errors, but be careful as you might end up with duplicate rows.

——————————————————————————————–
To Improve performance

Export

• Set the BUFFER parameter to a high value (e.g. 2Mb — entered as an integer “2000000″)
• Set the RECORDLENGTH parameter to a high value (e.g. 64Kb — entered as an integer “64000″)
• Use DIRECT=yes (direct mode export)
• Stop unnecessary applications to free-up resources for your job.
• If you run multiple export sessions, ensure they write to different physical disks.
• DO NOT export to an NFS mounted filesystem. It will take forever.

Import

• Create an indexfile so that you can create indexes AFTER you have imported data. Do this by setting INDEXFILE to a filename and then import. No data will be imported but a file containing index definitions will be created. You must edit this file afterwards and supply the passwords for the schemas on all CONNECT statements.
• Place the file to be imported on a separate physical disk from the oracle data files
• Increase DB_CACHE_SIZE (DB_BLOCK_BUFFERS prior to 9i) considerably in the init$SID.ora file
• Set the LOG_BUFFER to a big value and restart oracle.
• Stop redo log archiving if it is running (ALTER DATABASE NOARCHIVELOG;)
• Create a BIG tablespace with a BIG rollback segment inside. Set all other rollback segments offline (except the SYSTEM rollback segment of course). The rollback segment must be as big as your biggest table (I think?)
• Use COMMIT=N in the import parameter file if you can afford it
• Use STATISTICS=NONE in the import parameter file to avoid time consuming to import the statistics
• Remember to run the indexfile previously created

———————————————————————————————–
Good practices

• Always take care about CHARSETS when you do export and import. Using the wrong ones can convert your data in a lossy manner. The best situation is when your source and destination database have the same character sets, so you can avoid completely any character conversion. You control this behavior by setting NLS_LANG environment variable appropriately. When not set properly you may see ‘Exporting questionable statistics’ messages.
• After doing an export, it is better to check your dump by doing an import with the parameter SHOW=Y. This checks the validity of your dump file.

——————————————————————————————Other considerations

• You may need to patch your Oracle client (where you are running exp/imp) to the same level as the Oracle server to prevent errors
• When importing large amounts of data consider dropping indexes prior to the import to speed up the process and re-creating them once the import is completed
• The amount of archive logs that may be created on a large import may fill up your disk
• On INSERT triggers will fire, consider whether these need to be disabled
• Increasing the RECORDLENGTH (max 65535) parameter can improve the length of time to perform an import/export as well as DIRECT=y for export

——————————————————————————-
Tablespace fragmentation occurs when objects are created, dropped, enlarged, and shrunk. The more often this occurs, the more fragmented the tablespace becomes. This causes free space to be broken into many separated island-like chunks of space. In addition, if an object has multiple extents, they could be spread out over different parts of the tablespace; this causes performance and maintenance problems. With many small blocks of free space instead of fewer large free spaces, you may be unable to create some objects that you otherwise could have. By defragmenting the tablespace, the data is reorganized so that all free space is put into one contiguous area, and each object’s extents are grouped next to each other.

• The Export and Import utilities can fix fragmentation in two ways:
  They can take a table with multiple extents and resize it into a table with one larger extent. This larger single extent will encompass the total size of all previous table extents.
• They can make all objects within the tablespace adjacent to each other with no island-like characteristics, while at the same time merging all the free space into one larger free space.

The easiest and most dramatic method fixes both fragmentation problems for the entire database. The database is shut down during the process. This forces all sessions against the database to stop (which is why it’s the most dramatic method).

Fixing a table’s storage parameter is fairly easy compared to the entire tablespace, as only one table is affected. To fix an entire tablespace, you’ll have to drop and recreate it, along with all the tables within it.

Fix both fragmentation problems quickly

• Export the entire database, specifying FULL=Y and COMPRESS=Y. FULL=Y exports the entire database; COMPRESS=Y changes the INITIAL storage parameter of each table in the database (if necessary) to fit within one extent each. For example, if a table has five 10M extents, the COMPRESS=Y specification changes the table to one 50M extent when imported.
• Drop and recreate the entire database.
• Import the entire database with the FULL=Y parameter.

Defragment just one tablespace (a less dramatic method)

• Determine which tables exist in the tablespace and export just those tables.
• Drop all the tables in the tablespace.
• Issue the ALTER TABLESPACE COALESCE command (or drop and recreate the tablespace). This will coalesce all the free space within the tablespace into one contiguous area.
• Import the tables. Oracle will create the tables adjacent to each other within the tablespace, and all free space will still be contiguous in one chunk of space.

——————————————————————————-

Some more examples

EXP
To dump a single schema to disk (we use the scott example schema here)

- Login to server which has an Oracle client

exp <user>/<password> FILE=scott.dmp OWNER=scott
To export specific tables to disk: Login to server which has an Oracle client

exp SYSTEM/password FIlE=expdat.dmp TABLES=(scott.emp,hr.countries)

The above command uses two users : scott and hr

exp <user>/<password> FILE=scott.dmp TABLES=(emp,dept)

the above is only for one user : scott

IMP
To import the full database exported in the example above.
imp SYSTEM/password FULL=y FIlE=dba.dmp

To import just the dept and emp tables from the scott schema
imp SYSTEM/password FIlE=dba.dmp FROMUSER=scott TABLES=(dept,emp)

To import tables and change the owner

imp SYSTEM/password FROMUSER=blake TOUSER=scott FILE=blake.dmp TABLES=(unit,manager)
To import just the scott schema exported in the example above:
imp <user>/<password> FIlE=scott.dmp

If you do not supply any parameters then you enter an interactive session as illustrated below. (Responses to prompts are shown in blue. from the prompt type imp and press enter

$>imp
Import: Release 9.2.0.6.0 – Production on Thu Mar 29 15:07:43 2007
Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.
Username: SYSTEM
Password: password
Connected to: Oracle9i Enterprise Edition Release 9.2.0.6.0 – Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.6.0 – Production
Import file: expdat.dmp > /mention/path/of/dumpFile/includingFileName.dmp
Enter insert buffer size (minimum is 8192) 30720> (press enter to accept default)
Export file created by EXPORT:V09.02.00 via conventional path
import done in US7ASCII character set and AL16UTF16 NCHAR character set
import server uses AL32UTF8 character set (possible charset conversion)
List contents of import file only (yes/no): no > press enter
Ignore create error due to object existence (yes/no): no > press enter
Import grants (yes/no): yes > press enter
Import table data (yes/no): yes > press enter
Import entire export file (yes/no): no > press enter or type no
Username: give the userName for which you want the data to be imported
Enter table(T) or partition(T:P) names. Null list means all tables for user
Enter table(T) or partition(T:P) name or . if done: press enter
. importing TST_001_V2′s objects into TST_001_V2

SQL*Loader

SQL*Loader is a high-speed data loading utility that loads data from external files into tables in an Oracle database. SQL*Loader accepts input data in a variety of formats, can perform filtering, and can load data into multiple Oracle database tables during the same load session. SQL*Loader is an integral feature of Oracle databases and is available in all configurations.

Commands and Parameters

SQL*Loader can be invoked in one of three ways:

sqlldr

sqlldr keyword=value [keyword=value ...]

sqlldr value [value ...]

Valid Keywords/Parameters:

To check which options are available in any release of SQL*Loader use this command:

sqlldr help=y

PLEASE NOTE:

In addition to being passed by keyword, parameters may also be passed by position. To do this, you simply list the values after the sqlldr command in the correct order. For example, the following two SQL*Loader commands yield identical results:

sqlldr system/manager profile.ctl profile.log

sqlldr userid=system/manager control=profile.ctl log=profile.log

You can even mix the positional and keyword methods of passing command-line parameters. The one rule when doing this is that all positional parameters must come first. Once you start using keywords, you must continue to do so. For example:

sqlldr system/manager control=profile.ctl log=profile.log

When you pass parameters positionally, you must not skip any. Also, be sure to get the order right. You must supply parameter values in the order.One may specify parameters by position before but not after parameters specified by keywords. For example, system/manager control=profile.ctl log=profile.log is allowed, but system/manager control=profile.ctl profile.log is not, even though the position of the parameter ‘log’ is correct.

Parameter Precedence

The term “command-line” notwithstanding, most SQL*Loader command-line parameters can actually be specified in three different places:

• On the command line
• In a parameter file, which is then referenced using the PARFILE parameter
• In the control file

Parameters on the command line, including those read in from a parameter file, will always override values specified in the control file. In the case of the bad and discard file names, though, the control file syntax allows for each distinct input file to have its own bad file and discard files. The command line syntax does not allow for this, so bad file and discard file names specified on the command line only apply to the first input file. For any other input files, you need to specify these bad and discard file names in the control file or accept the defaults.

The FILE parameter adds a bit of confusion to the rules stated in the previous paragraph. As with the bad file and discard file names, you can have multiple FILE values in the control file. However, when you specify the FILE parameter on the command line, it does override any and all FILE values specified in the control file.

Parameters read from a parameter file as a result of using the PARFILE parameter may override those specified on the command line. Whether or not that happens depends on the position of the PARFILE parameter with respect to the others. SQL*Loader processes parameters from left to right, and the last setting for a given parameter is the one that SQL*Loader uses.

The SQL*Loader Environment

When we speak of the SQL*Loader environment, we are referring to the database, the SQL*Loader executable, and all the different files that you need to be concerned with when using SQL*Loader. These are shown in Figure 1-1.

Figure 1-1. The SQL*Loader environment

The functions of the SQL*Loader executable, the database, and the input data file are rather obvious. The SQL*Loader executable does the work of reading the input file and loading the data. The input file contains the data to be loaded, and the database receives the data.

Although Figure 1-1, doesn’t show it, SQL*Loader is capable of loading from multiple files in one session. When multiple input files are used, SQL*Loader will generate multiple bad files and discard files–one set for each input file.

File Types

SQL*Loader Control File
The control file is a text file written in a language that SQL*Loader understands. The control file tells SQL*Loader where to find the data (Input Data File), how to parse and interpret the data, and where to insert the data.

One can load data into an Oracle database by using the sqlldr (sqlload on some platforms) utility. Invoke the utility without arguments to get a list of available parameters. Look at the following example:

sqlldr username@server/password control=loader.ctl

sqlldr username/password@server control=loader.ctl

This sample control file (loader.ctl) will load an external data file containing delimited data:

load data

 infile 'c:\data\mydata.csv'

 into table emp

 fields terminated by "," optionally enclosed by '"'

 ( empno, empname, sal, deptno )

The mydata.csv file may look like this:

10001,"Scott Tiger", 1000, 40

10002,"Frank Naude", 500, 20

Another Sample control file with in-line data formatted as fix length records. The trick is to specify “*” as the name of the data file, and use BEGINDATA to start the data section in the control file:

load data

 infile *

 replace

 into table departments

 (  dept     position (02:05) char(4),

    deptname position (08:27) char(20)

 )

begindata

COSC  COMPUTER SCIENCE

ENGL  ENGLISH LITERATURE

MATH  MATHEMATICS

POLY  POLITICAL SCIENCE

Input Data and Datafiles
SQL*Loader reads data from one or more files specified in the control file. From SQL*Loader’s perspective, the data in the datafile is organized as records. A particular datafile can be in fixed record format, variable record format, or stream record format. The record format can be specified in the control file with the INFILE parameter. If no record format is specified, the default is stream record format.

If data is specified inside the control file (that is, INFILE * was specified in the control file), then the data is interpreted in the stream record format with the default record terminator.

LOBFILEs

LOB data can be lengthy enough that it makes sense to load it from a LOBFILE. LOB data instances are still considered to be in fields, but these fields are not organized into records. Therefore, the processing overhead of dealing with records is avoided. This type of or organization of data is ideal for LOB loading.

Bulk Loads

You can use SQL*Loader to bulk load objects, collections, and LOBs. SQL*Loader supports the following bulk loads:

• Two object types: column objects and row objects
• Load data from multiple datafiles during the same load session
• Two collection types: nested tables and VARRAYS
• Four LOB types: BLOBs, CLOBs, NCLOBs, and BFILEs.

Load Methods

SQL*Loader provides three methods to load data: Conventional Path, Direct Path, and External Table.

Conventional Path Load
Conventional path load builds an array of rows to be inserted and uses the SQL INSERT statement to load the data. During conventional path loads, input records are parsed according to the field specifications, and each data field is copied to its corresponding bind array. When the bind array is full (or no more data is left to read), an array insert is executed.

Direct Path Load
A direct path load builds blocks of data in memory and saves these blocks directly into the extents allocated for the table being loaded. A direct path load uses the field specifications to build whole Oracle blocks of data, and write the blocks directly to Oracle datafiles, bypassing much of the data processing that normally takes place. Direct path load is much faster than conventional load, but entails some restrictions.

A parallel direct path load allows multiple direct path load sessions to concurrently load the same data segments. Parallel direct path is more restrictive than direct path.

External Table Load
An external table load creates an external table for data in a datafile and executes INSERT statements to insert the data from the datafile into the target table.

There are two advantages of using external table loads over conventional path and direct path loads:

• An external table load attempts to load datafiles in parallel. If a datafile is big enough, it will attempt to load that file in parallel.
• An external table load allows modification of the data being loaded by using SQL functions and PL/SQL functions as part of the INSERT statement that is used to create the external table.

SQL*Loader is flexible and offers many options that should be considered to maximize the speed of data loads.  These include:

1. Use Direct Path Loads – The conventional path loader essentially loads the data by using standard insert statements.  The direct path loader (direct=true) loads directly into the Oracle data files and creates blocks in Oracle database block format.  The fact that SQL is not being issued makes the entire process much less taxing on the database.  There are certain cases, however, in which direct path loads cannot be used (clustered tables).  To prepare the database for direct path loads, the script $ORACLE_HOME/rdbms/admin/catldr.sql.sql must be executed.
2. Disable Indexes and Constraints. For conventional data loads only, the disabling of indexes and constraints can greatly enhance the performance of SQL*Loader.
3. Use a Larger Bind Array.  For conventional data loads only, larger bind arrays limit the number of calls to the database and increase performance.  The size of the bind array is specified using the bindsize parameter.  The bind array’s size is equivalent to the number of rows it contains (rows=) times the maximum length of each row.
4. Use ROWS=n to Commit Less Frequently. For conventional data loads only, the rows parameter specifies the number of rows per commit.  Issuing fewer commits will enhance performance.
5. Use Parallel Loads. Available with direct path data loads only, this option allows multiple SQL*Loader jobs to execute concurrently.
   • $ sqlldr control=first.ctl  parallel=true direct=true
   • $ sqlldr control=second.ctl parallel=true direct=true
6. Use Fixed Width Data. Fixed width data format saves Oracle some processing when parsing the data.  The savings can be tremendous, depending on the type of data and number of rows.
7. Disable Archiving During Load. While this may not be feasible in certain environments, disabling database archiving can increase performance considerably.
8. Use unrecoverable. The unrecoverable option (unrecoverable load data) disables the writing of the data to the redo logs.  This option is available for direct path loads only.

Using the table table_with_one_million_rows, the following benchmark tests were performed with the various SQL*Loader options. The table was truncated after each test.

Table 1.1 – Results indicate conventional path loads take longest.

The results above indicate that conventional path loads take the longest.  However, the bindsize and rows parameters can aid the performance under these loads.  The test involving the conventional load didn’t come close to the performance of the direct path load with the unrecoverable option specified.

It is also worth noting that the fastest import time achieved for this table (earlier) was 67 seconds, compared to 41 for SQL*Loader direct path – a 39% reduction in execution time.  This proves that SQL*Loader can load the same data faster than import.

These tests did not compensate for indexes.  All database load operations will execute faster when indexes are disabled.

Possible Issues while loading the data

Recovery from Failure

There are really only two fundamental ways that you can recover from a failed load. One approach is to delete all the data that was loaded before the failure occurred, and simply start over again. Of course, you need to fix whatever caused the failure to occur before you restart the load.

The other approach is to determine how many records were loaded successfully, and to restart the load from that point forward. Regardless of which method you choose, you need to think things through before you start a load.

Deleting data and restarting a load from scratch really doesn’t require any special functionality on the part of SQL*Loader. The important thing is that you have a reliable way to identify the data that needs to be deleted. SQL*Loader does, however, provide support for continuing an interrupted load from the point where a failure occurred. Using the SKIP command-line parameter or the SKIP clause in the control file, you can tell SQL*Loader to skip over records that were already processed in order to have the load pick up from where it left off previously.

Transaction Size

Transaction size is an issue related somewhat to performance, and somewhat to recovery from failure. In a conventional load, SQL*Loader allows you to specify the number of rows that will be loaded between commits. The number of rows that you specify has a direct impact on the size of the bind array that SQL*Loader uses, and consequently on the amount of memory required for the load. The bind array is an area in memory where SQL*Loader stores data for rows to be inserted into the database. When the bind array fills, SQL*Loader inserts the data into the table being loaded, and then executes a COMMIT.

The larger the transaction size, the more data you’ll need to reprocess if you have to restart the load after a failure. However, that’s usually not a significant issue unless your bind array size is quite large. Transaction size can also affect performance. Generally, the more data loaded in one chunk the better. So a larger bind array size typically will lead to better performance. However, it will also lead to fewer commits, resulting in the use of more rollback segment space.

Data Validation

Data validation is always a concern when loading data. SQL*Loader doesn’t provide a lot of support in this area, but there are some features at your disposal that can help you ensure that only good data is loaded into your database.

The one thing that SQL*Loader does do for you is ensure that the data being loaded into a column is valid given the column’s datatype. Text data will not be loaded into NUMBER fields, and numbers will not be loaded into DATE fields. This much, at least, you can count on. Records containing data that doesn’t convert to the destination datatype are rejected and written to the bad file.

SQL*Loader allows you to selectively load data. Using the WHEN clause in your SQL*Loader control file, you can specify conditions under which a record will be accepted. Records not meeting those conditions are not loaded, and are instead written to the discard file.

Finally, you can take advantage of the referential integrity features built into your database. SQL*Loader won’t be able to load data that violates any type of primary key, unique key, foreign key, or check constraint.

TIP: You don’t always have to rely on SQL*Loader’s features for data validation. It’s entirely feasible to load data into a staging table, run one or more external programs to weed out any rows that are invalid, and then transfer that data to a production table.

Data Transformation

Wouldn’t it be great if the data we loaded was always in a convenient format? Unfortunately, it frequently is not. In the real world, you may deal with data from a variety of sources and systems, and the format of that data may not match the format that you are using in your database. Dates, for example, are represented using a wide variety of formats. The date 1/2/2000 means one thing in the United States and quite another in Europe.

For dates and numbers, you can often use Oracle’s built-in TO_DATE and TO_NUMBER functions to convert a character-based representation to a value that can be loaded into a database DATE or NUMBER column. In fact, for date fields, you can embed the date format into your control file as part of the field definition.

SQL*Loader allows you access to Oracle’s entire library of built-in SQL functions. You aren’t limited to just TO_DATE, TO_NUMBER, and TO_CHAR. Not only can you access all the built-in SQL functions, you can also write PL/SQL code to manipulate the data being loaded.

Checklist before starting Sql*Loader

Check Environment Variables

The first and most important thing to take into a account is setting the environment variables correctly before starting Sql*Loader. Check the value of the environment variables at the machine from where you are starting your Sql*Loader session.

How to check your environment variables (e.g. NLS_LANG):

- UNIX
$ env | grep NLS_LANG
- Windows
Start –> Run –> regedit –> HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\<HOMEx>\NLS_LANG

1) Check NLS_LANG setting

Set NLS_LANG to the desired territory and characterset to prevent wrong data to be loaded or getting errors due to this data.

Syntax: NLS_LANG=<Language_Territory.Characterset>

Example: NLS_LANG=”Dutch_The_Netherlands.WE8ISO8859P15″

Possible problems due to an incorrect value:

- ORA-1722 invalid number

Reason:

NLS_NUMERIC_CHARACTERS is equal to ‘,.’ or ‘.,’ based upon the Territory (NLS_LANG) you are in.

Example:

The NLS_LANG setting above results in ‘,.’ For NLS_NUMERIC_CHARACTERS.

The number 13.4 is not valid in this case and produces ORA-1722.

- Characters not loaded correctly due to an incorrect characterset specified.

Reason:

The characterset specified needs to be the characterset of the data to be loaded (unless the CHARACTERSET keyword is used).

2) Check ORACLE_HOME:

Set above environment variables for the desired ORACLE_HOME from where you want SQL*Loader to be started.

Example: ORACLE_HOME=/u01/app/oracle/product/9.0.1

In Windows you can set your primary ORACLE_HOME using the Home Selector:

Start –> Programs –> Oracle Installation Products –> Home Selector

Possible problem due to an incorrect value:

- ORA-12560: TNS: protocol adapter error

Reason:

The SQL*Net connection fails because of the mixed environments

3) Check LD_LIBRARY_PATH (Unix only):

Check whether $ORACLE_HOME/lib is included in LD_LIBRARY_PATH.

Example: LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH

Possible problem:

- libwtc8 library cannot be found.

4) Check ORA_NLS33 (or ORA_NLS32) setting

Example: ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data

Possible problem due to an incorrect value:
- Segmentation Fault; Core Dump

Reason:
ORA_NLS33 points to the $ORACLE_HOME of another installation.
For example, the ORACLE_HOME where your Developer software is installed.

Limits / Defaults

1) Check the field lengths of the data to be loaded

Specify a length for the fields defined in the controlfile based upon the data to be loaded. Also check if the data to be loaded that it fits in the table columns specified.

A variable length field defaults to 255 bytes for a CHAR. If no datatype is specified, it defaults to a CHAR of 255 bytes as well.

See the ‘Utilities Manual’ chapter ‘SQL*Loader Control File Reference’ (see references) part ‘Calculating the Size of Field Buffers’ or related.

Possible errors are:

2) Check the datafile File Size Limit on your Operating System (Unix only)

On Unix, the filesize is limited by the shell’s filesize limit.
Set the limit of your filesize with ulimit (ksh and sh) or limit
(csh) command to a value larger than the size of your sqlloader datafile.

Editor’s note: Much of the content has been extracted from Oreilly and Oracle documents. Educational purpose only.