Debugging Oracle ESS portlet Producer/Consumer registration errors

When working with Fusion Apps ESS, you may encounter errors during job submissions that has to do with remote rendering of job parameters. ESS i.e. EssCentralUiApp utilizes portlet technology to remotely invoke a business view that is implemented as an Schedule Request Submission(SRS) ADF task flow.

In such cases, the ess server diagnostics logs will contain exception stack trace like below:

Caused by: javax.naming.NameNotFoundException; remaining name 'ESS_SRS_PortletProducerName_<webModuleName>_<hostname>_com'
        at oracle.adf.share.jndi.ContextImpl.findObject(ContextImpl.java:647)
        at oracle.adf.share.jndi.ContextImpl.lookup(ContextImpl.java:144)
        at oracle.adf.share.jndi.ContextImpl.lookup(ContextImpl.java:149)
        at javax.naming.InitialContext.lookup(InitialContext.java:392)
        at oracle.portlet.client.techimpl.wsrp.WSRPClientUtils.getProducerConnection(WSRPClientUtils.java:118)
        at oracle.portlet.client.techimpl.wsrp.WSRPClientUtils.getProducerConnection(WSRPClientUtils.java:98)
======================================================================

oracle.apps.fnd.applcp.srs.model.exception.PortletProducerRegistrationException: Fail to lookup producer id with dynamic registration
       at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.PortletRegistrationServiceAMImpl.lookupPortletProducerIdEx(PortletRegistrationServiceAMImpl.java:187)
        at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.PortletRegistrationServiceAMImpl.lookupPortletProducerId(PortletRegistrationServiceAMImpl.java:133)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequestSwitcher.computeTaskFlowId(ScheduleRequestSwitcher.java:105)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequestSwitcher.<init>(ScheduleRequestSwitcher.java:43)
        at sun.reflect.GeneratedConstructorAccessor1056.newInstance(Unknown Source)"
=======================================================================  
These errors typically indicate that the portlet producer registration has succeeded and there is some problem on the Portlet Binding side or the dynamic portlet producer registration on the EssCentralUiApp side has failed and there is some problem getting a valid portlet instance id.  Usually, you can get past these errors by manually deleting the portlet producer registration entry to force ESS to dynamically register the producer again.

Remember, the portlet producer lookup for dynamic registration is based on the specific name derived from value of the ESS jobDefinition property 'EXT_PortletContainerWebModule'. One can either follow these steps to delete the producer associated with EssCentralUiApp from Enterprise Manager (EM) or use WLST commands.

1. Login to EM console of ESS Server associated with Common Domain
2. Navigate to the WebCenter Service Configuration page as shown 

3. Select “portlet producer” and inspect the list of registered portlet producers 
4. Look for the offending webmodule entry from the exception stack; you should see only one entry
5. Delete the producer entry 
    (sometimes bouncing the server may help see the changes take effect)

In other cases, you may hit errors that are caused due to a stale WebService connection entry that needs to be removed. Exceptions like
.........
[ess_server1] [ERROR] [] [oracle.apps.fnd.applcp.srs]
....[APP: EssCentralUiApp]
.. [APPS_SESSION_ID: F145330E505C6A13E0436810C10A43B7] Register Producer failed due to errors.
@ Connection exists:
@ ESS_SRS_PortletProducerName_<webModuleName>_<hostname>_com-wsconn
        at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.PortletRegistrationServiceAMImpl.lookupPortletProducerIdEx(PortletRegistrationServiceAMImpl.java:187)
        at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.PortletRegistrationServiceAMImpl.lookupPortletProducerId(PortletRegistrationServiceAMImpl.java:133)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequestSwitcher.computeTaskFlowId(ScheduleRequestSwitcher.java:105)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequestSwitcher.<init>(ScheduleRequestSwitcher.java:43)
        at sun.reflect.GeneratedConstructorAccessor1056.newInstance(Unknown Source)"

Here, you can go through the System MBean browser to delete the portlet producer WS Connection entry from EM.

1. Login to EM in the CommonDomain and launch the System MBean Browser.     
2. Navigate to "ADFConnections" MBean
 Application Defined MBeans -> oracle.adf.share.connections -> Server: ess_server1 -> Application: EsscentralUiApp -> ADFConnections -> ADFConnections

3. Expand the "WebServiceConnection" node and verify that the stale connection 'ESS_SRS_PortletProducerName_<webModuleName>_<hostname>_com-wsconn' exists
4. Navigate back to the "Operations" tab and click on the 'removeConnection' operation.

After deleting the WS Connection entry, you may also want to delete the portlet producer registration entry as an extra measure.

For more information, refer the Fusion Applications Administrator's Guide chapter on "Troubleshooting Oracle WebCenter Portlets"

Oracle ESS Central UI Application and remote rendering of Job Parameters

Thought of sharing some information on how ESS Submission interface is designed to accept job parameters as user input.

ESS Global Submission UI is the primary interface for business users to submit job requests. The Central Submission interface is hooked up to Navigator menu in the Fusion Applications UI Shell [ Navigator -> Tools -> Schedule Processes].The extended functionality of ESS Global Submission UI allows users to submit job requests across CRM/HCM/FSCM domain in the Fusion Apps (FA) topology.

ESS CentralUi application deployed to the FA Common Domain acts as the portlet consumer and dynamically registers portlet producers at runtime. In the FA context, these portlet producers are basically remote web applications (deployed across Financials/CRM/HCM domain weblogic managed servers), that host the parameters ViewObject (VO) to facilitate the collection of user input as parameters for ESS job request runtime execution.

The Oracle Portlet Bridge exposes JSF applications and task flows as JSR 168 portlets. In Oracle Fusion Applications, portlets are WSRP portlets. The purpose of the Web Services for Remote Portlets (WSRP) protocol is to provide a web services standard that allows for the visual "plug-n-play" of remote running portlets from disparate sources.

Just as a web application relies on the servlet container to invoke custom developed servlets, a portal application relies on the portal container (i.e. Oracle Webcenter implementation) that uses the portlet API to invoke portlets. The ESS CentralUi application contacts the portlet producers that provide the portlets (i.e the ESS SRS TaskFlow with parametersVO exposed as portlet by the portlet-bridge) to be rendered on the ESS job submission page. The figure below illustrates the basic architecture of the ESS consumer web application EssCentralUiApp' interaction with the portlet producers.

Note: The portlet producer look-up for dynamic registration is based on the specific name derived from value of the ESS jobDefinition property 'EXT_PortletContainerWebModule'.

Refer the official Oracle Webcenter documentation for more details on Portlets & Portlet-bridge implementation.

Java 2D-Array Spiral Order Traversal Implementation

Recently, I came across a problem that required iterating through the elements of the 2D array in a clockwise inward spiral starting from the top left cell of the matrix. Basically we have a 2d array, and the requirement is to represent it with a 1D array printing the elements in a spiral order (as in example below).

1	2	3	4	5
14	15	16	17	6
13	20	19	18	7
12	11	10	9	8

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Am sure there is a much better and efficient way to solve this problem. Here is my version of the implementation (recursion approach) that works for any n*n matrix. Details in the self-explanatory Java code:

Re-wire Oracle SOA Datasource to use Active GridLink for RAC

Starting 11gR1 PS5 (11.1.1.6), Oracle SOA Suite platform is enhanced to support Active GridLink for Real Application Clusters (RAC).

WebLogic Server Active GridLink (AGL) for RAC provides the best available support for the RAC features in Oracle Database 11g. The generic data source is the implementation for single database access. The Multi-data source is the native WebLogic middle tier implementation for RAC integration, which does not leverage Oracle Notification Service (ONS).

The GridLink data source is the new Active GridLink implementation and includes the features of generic data sources, taking advantage of Oracle RAC supporting Fast Connection Failover (FCF), Runtime Connection Load-Balancing (RCLB), and XA Affinities. A GridLink data source is created for JDBC connectivity between WebLogic Server and a service targeted to an Oracle RAC cluster. It uses the Oracle Notification Service (ONS) to adaptively respond to state changes in an Oracle RAC instance.

Here are couple of good resources that can help you understand the AGL concepts in more detail:
- What is Active GridLink
- Benefits of AGL data sources

Basically, a data source configuration is always an Active GridLink (AGL) data source configuration if FanEnabled=true or the OnsNodeList is non-null.

Here is a modified version of a python script (borrowed from my colleague), that you can use to rewire the SOA data sources to use Active GridLink for RAC databases. Before executing the script, remember to shutdown all the WLS SOA domain servers; its an offline wlst script.

Creating logical partition on Exalogic X3-2 ECHO vServer using LVM

I switched to using the recently announced Exalogic X3-2 Echo (2.0.6.0.0) guest template for packaging the application software VM. However, on trying to increase the disk space as in the Navstar (2.0.4.0.0) template version using the modifyjeos tool (version 1.1.0-17), I ran into the the following error:

# modifyjeos -f System.img -P App.img 6144 /u01 -S 2048 -n EL_ECHO_MYAPP_2.0_VM_TEMPLATE
Starting VM image reconfiguration...
ERROR: Unknow filesystem.

No detailed messages other than the single line below were logged to /var/log/modifyjeos.log
       ...modifyjeos version 1.1.0-17 starting
 
I later learnt that modifyjeos does not support LVM and this is one of the new feature in Exalogic ECHO (2.0.6.0.0) release. Using Logical Volume Manager (LVM) is recommended to increase/create the virtual disk space. Here are the steps followed to prepare the Exalogic Echo guest VM instance for creating a logical partition to host the application software.


Refer the official Exalogic X3-2 Elastic Cloud Administrator's Guide "Managing LVM Partitions on Guest vServers" for more details on this subject

Configuring SOA Distributed Destination Topics using WLST

The organization of topics and the granularity of subscriptions to them is an important part of a publish/subscribe application’s architecture. A message delivered to a topic is distributed to all consumers that are subscribed to that topic. Ensuring that a clustered application gets exactly one copy of each message, and parallel processing of topics messages may require significant additional configuration, coding, and message hops. To address such limitations, WebLogic JMS provides load balancing and failover for physical JMS destination members (queues or topics) through its "Distributed Destinations" that applications can leverage.

In a typical SOA Suite production cluster deployment, load balancing the JMS topic messages against the SOA composites will be a common requirement. It is to ensure that messages are evenly distributed (load balanced) to SOA cluster members, and to avoid duplication in some scenarios. This can be achieved by configuring the SOA JMS Topic to use a "Distributed Destination" setting the appropriate forwarding policy and other advanced parameters.

Here is a sample WLST offline script:


Note that there in no support to define the UniformDistributed Queue/Topic in weblogic application extension templates' file config/jms/<%jms-module%>-jms.xml (as of the latest SOA PS6 release). Refer the "Fusion Middleware Programming JMS for Oracle WebLogic Server" guide for more details on WLS JMS Distributed Destinations.

Creating Application VM Templates for Exalogic X3-2

 It is a well known fact that one of the primary goal for applications virtualization is to provide enterprise users with a consistent and reliable experience through accelerated deployment. For complex enterprise-class applications like Oracle SOA Suite (or IdM/any other business software), virtualization facilitates rapid provisioning and delivery of optimized application configuration to meet security, high availability and performance standards.

Exalogic Elastic Cloud X3-2 is Oracle' flagship engineered system for running business applications in a highly performant environment. The Exalogic X3-2 software includes a optimized Exabus­ enabled Type I Xen hypervisor i.e. Oracle VM Server, which allows to host virtual servers running Oracle Linux. In this post, I will share my experience creating an application template for deploying on Exalogic.

The application template creation procedure for Exalogic can be split into 2 broad categories:

1. Preparing the Oracle VM Server environment
2. Creating the application VM template

Note that access to an Exalogic instance is not required for the purpose of creating the application templates.You will require the following software though; download from eDelivery.oracle.com

• "Oracle Linux JeOS for Building Oracle VM Templates Media Pack v13 for x86_64 (64 bit)"
• "Oracle Exalogic 2.0.4.0.0 Base Guest Template" for Exalogic Linux x86-64 (64 bit) V35512-01  (requires OVM Server 3.x or later)

First, the steps to configure the Oracle VM Server:

1) Connect to Oracle VM Server 3.x
- it is recommended to use VM Server 3.x installation or later
- ssh username@hostname

2) Configure xend-config.sxp on the OVM Server (OVS) to use bridge networking
- the OVS system must be setup so that we can attach virtual machines to the external network. This is done by creating a virtual switch within dom0 (domain zero i.e. management domain in a OVS env responsible for hardware detection) that takes packets from the virtual machines and forwards them onto the physical network so they can see the internet and other machines on your network. The piece of software we use to do this is called the Linux bridge and its core components reside inside the Linux kernel. In this case the “bridge” is effectively our virtual switch.

# vi /etc/xen/xend-config.sxp
(network-script network-bridge)
(vif-script vif-bridge)

3) Restart OVM Server
# [root@ovmserver BASE]# reboot

4) Configure your OVM Server operating system to use a public Yum repository
- download and install any prerequisite Oracle Enterprise Linux packages not installed on your computer from the public Oracle Yum repository

--- Oracle Linux 4, Update 6 or Newer
# cd /etc/yum.repos.d
# mv Oracle-Base.repo Oracle-Base.repo.disabled
# wget http://public-yum.oracle.com/public-yum-el4.repo

--- Oracle Linux 5
# cd /etc/yum.repos.d
# wget http://public-yum.oracle.com/public-yum-el5.repo

--- Oracle Linux 6
# cd /etc/yum.repos.d
# wget http://public-yum.oracle.com/public-yum-ol6.repo

5) Install the JeOS dependent packages with the following command:
# yum install createrepo

6) Install "Oracle Linux JeOS for Building Oracle VM Templates Media Pack v13 for x86_64 (64 bit)"
     - JeOS is particularly useful and handy when you need to re-size the VM template Root Disk Image
- download the zip files for JeOS base and JeOS base image for required operating system to your OVS system in local directory e.g /u01/JeOS.
        (you will have following rpms in your local directory after uncompressing the zip files)
ovm-modify-jeos-1.1.0-17.el5.noarch.rpm
ovm-template-config-1.1.0-8.el5.noarch.rpm
ovm-el5u8-xvm-jeos-1.1.0-2.el5.x86_64.rpm

- install these rpms

# rpm -ivh ovm-modify-jeos-1.1.0-17.el5.noarch.rpm
# rpm -ivh ovm-template-config-1.1.0-8.el5.noarch.rpm
# rpm -ivh ovm-el5u8-xvm-jeos-1.1.0-2.el5.x86_64.rpm

7) At the end of this step, you will have the base OVM Server environment ready to create the virtual machines (or virtual host servers in Exalogic terminology).

Next, the steps to create the application VM template. Three main phases:

1. Creating VM using Exalogic Base Guest template  
2. Installing and configuring the required application software
3. Preparing the VM template TAR

Phase1

1. From eDelivery.oracle.com download "Oracle Exalogic 2.0.4.0.0 Base Guest Template" for Exalogic Linux x86-64 (64 bit) V35512-01 

2. Extract the Exalogic Guest Template contents to a directory location (say /Exalogic/BASE) on your Oracle VM Server 3.x installation

    # ssh username@ovshostname
     # mkdir -p /Exalogic/BASE
      
    # pwd
   /Exalogic/BASE
   
    BASE]# ls
    System.img  vm.cfg 
    
3. Change the path to the virtual disk in the vm.cfg file, as follows:
        disk = ['file:/OVS/seed_pool/OVM_simple_/System.img,hda,w'] to disk = ['file:/Exalogic/BASE/System.img,hda,w']  

4. In the Exalogic Guest template vm.cfg file, set vif = ['bridge=bond0'], where bond0 is the bridging Ethernet adapter on OVS. It is usually called 'xenbr0'

5. Start the VM with '-c' flag, and press Ctrl+D when it gets to the OVM template configuration stage
# cd /Exalogic/BASE
# xm create vm.cfg -c  
       (alternatively, you can skip the '-c' flag and connect to the VM using the command: xm console <vm-
         domain-Id>)

6. Login as 'root' to the created VM host (lets call it App-VM)
      - the default root password is 'ovsroot'

7. Add the network script ifcfg-eth0 in the App-VM 
- if eth0 already exists, use a different one
      - there are two options, DHCP, and Static IP Address. If you choose to make use of DHCP to automatically assign IP addresses to servers or virtual machines within a network, you must ensure that a DHCP server is set up and available within your VM environment

 # vi /etc/sysconfig/network-scripts/ifcfg-eth0
-----------------------------------------
DEVICE=eth0
            BOOTPROTO=static
            ONBOOT=yes
            IPADDR=10.200.250.300
            NETMASK=255.255.252.0
            GATEWAY=10.200.220.320
            DELAY=0

8. Configure the Network
- Edit the host name in the App-VM /etc/sysconfig/network file
- Edit the host name in /etc/hosts/
- Update /etc/resolv.conf as well

9. Turn off the initial configuration of the OVM template
# vi /etc/sysconfig/ovmd
set INITIAL_CONFIG=no

10. Use ifup to bring eth0 online or reboot VM
# /sbin/ifup eth0

11. Restart the networking service
# /etc/init.d/network restart

Phase2

1. Copy the necessary application software to the App-VM environment created in phase1
- you may need to configure the required JDK as well if your application depends on it. Exalogic guest template OS does not package any Java runtime
2. Install and configure the business applications in the App-VM

Phase3

1. Prepare to create a Template from the running App-VM
- Update ovmd script

# vi /etc/sysconfig/ovmd
Set the "INITIAL_CONFIG" parameter to "yes"
INITIAL_CONFIG=yes

2. Cleanup App-VM host specific OS-network level configuration
- Remove DNS information
    - Remove extraneous SSH information
    - Cleanup network file
# cd /etc/sysconfig
# sed -i '/^GATEWAY/d' network
    - Cleanup hosts file
    - Remove networking information
# cd /etc/sysconfig/network-scripts
# rm -f ifcfg-*eth*
# rm -f ifcfg*ib*
# rm -f ifcfg*bond*
     
# cd /etc/sysconfig/networking/profiles/default
# rm -f ifcfg-*eth*
# rm -f ifcfg*ib*
# rm -f ifcfg*bond*
    - Remove logs
    - Remove kernel messages
# cd /var/log
# rm -f dmesg
# dmesg -c
    - Remove history
# rm -f /root/.bash_history
# history -c

(Note: execute steps 3 through 5 below from the OVM Server host)

3. Shutdown App-VM

# xm list
# xm shutdown <VM_ID>

4. Compact the App-VM template vDisks (optional)

# [root@ovmserver BASE] modifyjeos -f System.img -zero-out-all 

5. Package the vServer with its virtual disk

# tar -pczvf <app_template_name>.tgz <el_base_template_directory>

The TAR VM template <app_template_name>.tgz is now ready to be uploaded/deployed to the Exalogic Elastic Cloud

Hope this helps to get a general idea on what it takes to create application templates on Linux for Exalogic X3-2 system. Feel free to share your experience/comments/tips .