Quick Setup of OpenStack Icehouse Development Env on Ubuntu14

OpenStack - there are quite a lot of vendors (including large enterprises) gravitating around the OpenStack ecosystem. It is a good thing that there is ample competition, which means more options for customers and hopefully better business models driving the cloud enterprise market. I look forward to research more from a technical standpoint.

In my view, a good starting point to learn more about the different technical features in OpenStack, is to experiment with them on a  small scale local environment. My test bed is a Ubuntu 14.4 64-bit Linux system on a 8x Intel Xeon CPU machine, with 8G memory and two physical network interface cards. The goal is to basically setup an all-in-one configuration, where all the services, including compute services, are installed on the same node. A controller node is where most of the OpenStack services are configured, and will be installed on my Ubuntu system.

Here, I will discuss couple of quick OpenStack development environment setup options:
    a) Using stable Git Icehouse repository
                OR
    b) Using Vagrant Box

Option-A:  Deploy OpenStack IceHouse using Git repo

1. Create work directory for OpenStack project, say $ICE_STACK_DIR
 
    # mkdir /scratch/<user>/icehouse  

2. Clone stable/icehouse git repository

   - used Netbeans ide to clone the Git repository branch to workdir location; alternately, run the following command from $ICE_STACK_DIR
 
    # git clone -stable/icehouse https://github.com/openstack-dev/devstack.git
   
3. Modify Devstack configuration file to override default settings as needed

  - localrc is a user-maintained settings file used to configure DevStack. It is deprecated and has been replaced by local.conf. More details here

  Sample local.conf: 



4. Install DevStack as a non-root user

   # cd $ICE_STACK_DIR/devstack
   # ./stack.sh

  - Read more about the stack.sh script in the official documentation

 - The default services configured by DevStack are Identity (Keystone), Object Storage (Swift), Image Storage (Glance), Block Storage (Cinder), Compute (Nova), Network (Neutron), Dashboard (Horizon)

  - During install run, hit errors like below:
        cp: cannot create regular file '/etc/nova/policy.json': Permission denied
     
    the resolution was to basically edit the file work-dir/devstack/lib/nova and change to 'sudo cp' for the failing file-access occurrences; for e.g. the following changes were made

        sudo cp -p $NOVA_DIR/etc/nova/policy.json $NOVA_CONF_DIR

        # Get the sample configuration file in place
        sudo cp -p $NOVA_DIR/etc/nova/api-paste.ini $NOVA_CONF_DIR
        sudo chown $STACK_USER $NOVA_CONF_DIR

   - To give an estimate, deploying DevStack in  my environment took between 5-6 minutes (after prior multiple failed attempts)

5. Perform basic sanity tests

   - Run the test scripts
         # cd $ICE_STACK_DIR/devstack/tests
         # ./functions.sh
         # ./test_config.sh
   
   - Run the exercise scripts
        # cd $ICE_STACK_DIR/devstack/exercises
   
       # ./horizon.sh 
   
        -- expect to see something like the following message printed on the console if everything goes well with the deployment
        .............
        + set +o xtrace
        *********************************************************************
        SUCCESS: End DevStack Exercise: ./horizon.sh
        *********************************************************************

6. Launch OpenStack Horizon Dashboard

   - Go to URL http://my.eth1.ipv4.address
   - Logon as default user 'demo' or 'admin' and password $ADMIN_PASSWORD set in local.conf

   Here is a screenshot of the dashboard System Info panel

7. Try creating instances from the Dashboard

   - Refer OpenStack Admin Guide for more details on managing the resources and services using the Horizon dashboard    

8.  Stopping and Restarting DevStack

   - To stop all processes that were started by stack.sh
        # cd $ICE_STACK_DIR/devstack
        # ./unstack.sh
   
   - To restart DevStack
        # cd $ICE_STACK_DIR/devstack
        # ./rejoin-stack.sh

Option-B. Deploy DevStack Icehouse using Vagrant

I found this blog article 'OpenStack Cloud Computing Cookbook' by Kevin very helpful in setting up my local development environment virtual machines.

Installed the following:

1. VirtualBox 4.3.10
2. Vagrant 1.4.3
3. Vagrant Cachier plugin    
     
and then followed the instructions as-is, they just work as documented.

Hoping to share my experiments with OpenStack as I learn more ....

Enterprise Architecture Balanced Scorecard

Enterprise Architecture (EA) is an interesting topic for those in the IT industry. There are tons of articles on the web that talks about the role of the Enterprise Architect, and how this IT-centric function  or responsibility is key to serve the entire business, including strategy and execution well beyond IT and IT solution delivery.

On the other hand, you may come across debates and lots of conflicting opinions on the differences in focus between an Enterprise Architect, a Business Architect, an Information Architect and a Process Architect. However, having spend considerable time in the enterprise software development space, it is my opinion that 'job titles' do not matter. As an Enterprise IT professional, I have learnt that sound practices like clearly defining the problem statement, asking the right/relevant questions, focus on priorities and a sense of humility will take us a long way in helping achieve the business/technical goals to serve the wider global community. Well, this is a complex subject and I have in no way figured it all. The best part is that it is a collaborative learning experience.

In this post, I will share my opinion on what 'Enterprise Architecture' means to me and use Balanced Scorecard as a conceptual tool for illustrating Enterprise Architecture (EA) as a means for Business-IT Alignment. This approach connects business model to EA vision and proposes design methods/perspectives to be measured in the Business, Applications, Data and Infrastructure areas.

Remote Profiling Oracle SOA application server using NetBeans7.4

In this section, I will share my experience profiling the Oracle SOA Suite application server using NetBeans 7.4 from Windows7 desktop - where the WLS SOA managed server is running on a remote Enterprise Linux 5 machine. Here are the main steps:

A. Choose Profile > Attach Profiler (Attach Profile icon) from the main menu of Netbeans IDE to open the Attach Profiler dialog box.

Specify the location of the application and the connection method.

B. Before profiling, perform the following

Step 1: Make sure the target application is configured to run using Java 6+.

Step 2: If you have not done it before, create a Remote profiling pack (click the link in the IDE message window) for the selected OS & JVM and upload it to the remote system.

 - Download the profiling pack file 'profiler-server-linuxamd64.zip' that will be created and copy to the remote location. (extracted dir contents of the zip below)

profiler-server-linuxamd64
|- bin
|- lib
- README

Step 3: If you have not run profiling on the remote system yet, run the <remote>/bin/calibrate-16.sh script first to calibrate the profiler.

You should see something like

profiler-server-linuxamd64]$ ./bin/calibrate-16.sh 

Profiler Agent: JNI OnLoad Initializing...
Profiler Agent: JNI OnLoad Initialized successfully
Starting calibration...
Calibration performed successfully
For your reference, obtained results are as follows:
Approximate time in one methodEntry()/methodExit() call pair:
When getting absolute timestamp only: 2.0325 microseconds
When getting thread CPU timestamp only: 1.85 microseconds
When getting both timestamps: 3.84 microseconds

Approximate time in one methodEntry()/methodExit() call pair
in sampled instrumentation mode: 0.0425 microseconds

Step 4: Add the following parameter(s) to the SOA application startup script:

- Add JVM Options property to $DOMAIN_HOME/bin/startManagedWebLogic.sh

JAVA_OPTIONS="${JAVA_OPTIONS} -XX:+UseLinuxPosixThreadCPUClocks -agentpath:<remote>/lib/deployed/jdk16/linux-amd64/libprofilerinterface.so=<remote>/lib,5140"
export JAVA_OPTIONS

Step 5: Start the target application. The process will wait for the profiler to connect.

On starting the soa_server1, you should see something like this in the server diagnostics log
.........
Profiler Agent: Waiting for connection on port 5140 (Protocol version: 14)

Step 6: Submit to close the Attach Settings dialog

Step 7: Click the Attach button in the Attach Profiler dialog to connect to the target application server and resume its execution.

................
Profiler Agent: Established connection with the tool
Profiler Agent: Standard session
<Feb 11, 2014 3:04:43 PM PST> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.> 
<Feb 11, 2014 3:04:43 PM PST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.> 
<Feb 11, 2014 3:04:43 PM PST> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 24.45-b08 from Oracle Corporation.> 
<Feb 11, 2014 3:04:44 PM PST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.3.0.0  Wed Dec 11 11:57:28 PST 2013 1567342 > 
<Feb 11, 2014 3:04:47 PM PST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.> 

C. The IDE attaches to the remote JVM and you can view the profiling data of the remote SOA WLS managed server as for a local application

   - sample Snapshot of the VM Telemetry view displaying info on Classes loaded and running Threads

You can find more details about NetBeans7.4 profiling capabilities in the official NetBeans7.4 documentation chapter "Testing and Profiling Java Application Projects".

Debugging Oracle ESS portlet Producer/Consumer registration errors

When working with Fusion Apps ESS, you may encounter errors during job submissions that has to do with remote rendering of job parameters. ESS i.e. EssCentralUiApp utilizes portlet technology to remotely invoke a business view that is implemented as an Schedule Request Submission(SRS) ADF task flow.

In such cases, the ess server diagnostics logs will contain exception stack trace like below:

Caused by: javax.naming.NameNotFoundException; remaining name 'ESS_SRS_PortletProducerName_<webModuleName>_<hostname>_com'
        at oracle.adf.share.jndi.ContextImpl.findObject(ContextImpl.java:647)
        at oracle.adf.share.jndi.ContextImpl.lookup(ContextImpl.java:144)
        at oracle.adf.share.jndi.ContextImpl.lookup(ContextImpl.java:149)
        at javax.naming.InitialContext.lookup(InitialContext.java:392)
        at oracle.portlet.client.techimpl.wsrp.WSRPClientUtils.getProducerConnection(WSRPClientUtils.java:118)
        at oracle.portlet.client.techimpl.wsrp.WSRPClientUtils.getProducerConnection(WSRPClientUtils.java:98)
======================================================================

oracle.apps.fnd.applcp.srs.model.exception.PortletProducerRegistrationException: Fail to lookup producer id with dynamic registration
       at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.PortletRegistrationServiceAMImpl.lookupPortletProducerIdEx(PortletRegistrationServiceAMImpl.java:187)
        at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.PortletRegistrationServiceAMImpl.lookupPortletProducerId(PortletRegistrationServiceAMImpl.java:133)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequestSwitcher.computeTaskFlowId(ScheduleRequestSwitcher.java:105)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequestSwitcher.<init>(ScheduleRequestSwitcher.java:43)
        at sun.reflect.GeneratedConstructorAccessor1056.newInstance(Unknown Source)"
=======================================================================  
These errors typically indicate that the portlet producer registration has succeeded and there is some problem on the Portlet Binding side or the dynamic portlet producer registration on the EssCentralUiApp side has failed and there is some problem getting a valid portlet instance id.  Usually, you can get past these errors by manually deleting the portlet producer registration entry to force ESS to dynamically register the producer again.

Remember, the portlet producer lookup for dynamic registration is based on the specific name derived from value of the ESS jobDefinition property 'EXT_PortletContainerWebModule'. One can either follow these steps to delete the producer associated with EssCentralUiApp from Enterprise Manager (EM) or use WLST commands.

1. Login to EM console of ESS Server associated with Common Domain
2. Navigate to the WebCenter Service Configuration page as shown 

3. Select “portlet producer” and inspect the list of registered portlet producers 
4. Look for the offending webmodule entry from the exception stack; you should see only one entry
5. Delete the producer entry 
    (sometimes bouncing the server may help see the changes take effect)

In other cases, you may hit errors that are caused due to a stale WebService connection entry that needs to be removed. Exceptions like
.........
[ess_server1] [ERROR] [] [oracle.apps.fnd.applcp.srs]
....[APP: EssCentralUiApp]
.. [APPS_SESSION_ID: F145330E505C6A13E0436810C10A43B7] Register Producer failed due to errors.
@ Connection exists:
@ ESS_SRS_PortletProducerName_<webModuleName>_<hostname>_com-wsconn
        at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.PortletRegistrationServiceAMImpl.lookupPortletProducerIdEx(PortletRegistrationServiceAMImpl.java:187)
        at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.PortletRegistrationServiceAMImpl.lookupPortletProducerId(PortletRegistrationServiceAMImpl.java:133)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequestSwitcher.computeTaskFlowId(ScheduleRequestSwitcher.java:105)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequestSwitcher.<init>(ScheduleRequestSwitcher.java:43)
        at sun.reflect.GeneratedConstructorAccessor1056.newInstance(Unknown Source)"

Here, you can go through the System MBean browser to delete the portlet producer WS Connection entry from EM.

1. Login to EM in the CommonDomain and launch the System MBean Browser.     
2. Navigate to "ADFConnections" MBean
 Application Defined MBeans -> oracle.adf.share.connections -> Server: ess_server1 -> Application: EsscentralUiApp -> ADFConnections -> ADFConnections

3. Expand the "WebServiceConnection" node and verify that the stale connection 'ESS_SRS_PortletProducerName_<webModuleName>_<hostname>_com-wsconn' exists
4. Navigate back to the "Operations" tab and click on the 'removeConnection' operation.

After deleting the WS Connection entry, you may also want to delete the portlet producer registration entry as an extra measure.

For more information, refer the Fusion Applications Administrator's Guide chapter on "Troubleshooting Oracle WebCenter Portlets"

Oracle ESS Central UI Application and remote rendering of Job Parameters

Thought of sharing some information on how ESS Submission interface is designed to accept job parameters as user input.

ESS Global Submission UI is the primary interface for business users to submit job requests. The Central Submission interface is hooked up to Navigator menu in the Fusion Applications UI Shell [ Navigator -> Tools -> Schedule Processes].The extended functionality of ESS Global Submission UI allows users to submit job requests across CRM/HCM/FSCM domain in the Fusion Apps (FA) topology.

ESS CentralUi application deployed to the FA Common Domain acts as the portlet consumer and dynamically registers portlet producers at runtime. In the FA context, these portlet producers are basically remote web applications (deployed across Financials/CRM/HCM domain weblogic managed servers), that host the parameters ViewObject (VO) to facilitate the collection of user input as parameters for ESS job request runtime execution.

The Oracle Portlet Bridge exposes JSF applications and task flows as JSR 168 portlets. In Oracle Fusion Applications, portlets are WSRP portlets. The purpose of the Web Services for Remote Portlets (WSRP) protocol is to provide a web services standard that allows for the visual "plug-n-play" of remote running portlets from disparate sources.

Just as a web application relies on the servlet container to invoke custom developed servlets, a portal application relies on the portal container (i.e. Oracle Webcenter implementation) that uses the portlet API to invoke portlets. The ESS CentralUi application contacts the portlet producers that provide the portlets (i.e the ESS SRS TaskFlow with parametersVO exposed as portlet by the portlet-bridge) to be rendered on the ESS job submission page. The figure below illustrates the basic architecture of the ESS consumer web application EssCentralUiApp' interaction with the portlet producers.

Note: The portlet producer look-up for dynamic registration is based on the specific name derived from value of the ESS jobDefinition property 'EXT_PortletContainerWebModule'.

Refer the official Oracle Webcenter documentation for more details on Portlets & Portlet-bridge implementation.

Java 2D-Array Spiral Order Traversal Implementation

Recently, I came across a problem that required iterating through the elements of the 2D array in a clockwise inward spiral starting from the top left cell of the matrix. Basically we have a 2d array, and the requirement is to represent it with a 1D array printing the elements in a spiral order (as in example below).

1	2	3	4	5
14	15	16	17	6
13	20	19	18	7
12	11	10	9	8

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Am sure there is a much better and efficient way to solve this problem. Here is my version of the implementation (recursion approach) that works for any n*n matrix. Details in the self-explanatory Java code:

Re-wire Oracle SOA Datasource to use Active GridLink for RAC

Starting 11gR1 PS5 (11.1.1.6), Oracle SOA Suite platform is enhanced to support Active GridLink for Real Application Clusters (RAC).

WebLogic Server Active GridLink (AGL) for RAC provides the best available support for the RAC features in Oracle Database 11g. The generic data source is the implementation for single database access. The Multi-data source is the native WebLogic middle tier implementation for RAC integration, which does not leverage Oracle Notification Service (ONS).

The GridLink data source is the new Active GridLink implementation and includes the features of generic data sources, taking advantage of Oracle RAC supporting Fast Connection Failover (FCF), Runtime Connection Load-Balancing (RCLB), and XA Affinities. A GridLink data source is created for JDBC connectivity between WebLogic Server and a service targeted to an Oracle RAC cluster. It uses the Oracle Notification Service (ONS) to adaptively respond to state changes in an Oracle RAC instance.

Here are couple of good resources that can help you understand the AGL concepts in more detail:
- What is Active GridLink
- Benefits of AGL data sources

Basically, a data source configuration is always an Active GridLink (AGL) data source configuration if FanEnabled=true or the OnsNodeList is non-null.

Here is a modified version of a python script (borrowed from my colleague), that you can use to rewire the SOA data sources to use Active GridLink for RAC databases. Before executing the script, remember to shutdown all the WLS SOA domain servers; its an offline wlst script.