Monday, August 4, 2014

Calling external REST services supporting JSON messaging standard in Oracle SOA 11g

SOA Suite 11g doesn't provide an out of box way to call JSON based REST services. In this blog we shall the alternative options to achieve the same.

HTTP binding in SOA Suite 11g  has ability to make calls to endpoints based on HTTP verbs(GET, POST only) in request-response (or) one way modes as depicted below.
  



However Oracle's Documentation states that -"An outbound HTTP binding reference supports only XML as a response from an external HTTP endpoint. The response should contain the correct XML part name according to outbound expectations."

In other words Http binding supports only XML as output for either GET/POST. For input it supports only url-encoded and xml. Though the url-encoded input has to be expressed/defined in an xsd format as shown below.



 Another major hindrance is that - Majority of the REST services created in IT world use JSON rather than XML as their messaging standard dues to its light weight and simplicity ( the same reason to use REST as against SOAP). HTTP adapter/binding has no support for calling REST endpoints which use JSON as a message standard as against XML. 
Does this leave us no way to call REST services using JSON ? Well not really. We always have Java to fall back on to do something that can't be done ( or provided) in SOA Suite.

The below Java code can be used to call REST service using JSON. The input of the methods is an xml format represented in a string ( to make it easier to interact with bpel). The following are the key points of this code

a. Takes input in XML string format.
b. Converts the XML string to JSON object.
c. Creates a HTTPClient and makes a call to post/get method using the REST URI providing the required basic http authentication credentials.
d. Extra code to bypass the SSL layer ( only to be used in dev mode). In production one should import the right certificates to authenticate themselves.

RestService.java

package oracle.soa.rest;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.DefaultHttpClient;

import net.sf.json.JSON;
import net.sf.json.xml.XMLSerializer;

import org.apache.http.auth.AuthScope;
import org.apache.http.auth.Credentials;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.methods.HttpGet;

public class RestService
{
    public HttpResponse callPOSTService(String xmlStr,String url) throws ClientProtocolException, IOException
    {
          // Convert the input XML to the JSON String
   XMLSerializer xmlSerializer = new XMLSerializer();  
   JSON json = xmlSerializer.read(xmlStr);  
   System.out.println(json);
   // Call POST of REST Service with JSON input

                        DefaultHttpClient client = new DefaultHttpClient();
                        client = WebClientDevWrapper.wrapClient(client);

                        Credentials creds = new UsernamePasswordCredentials("username", "password");
                        client.getCredentialsProvider().setCredentials( new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT), creds);

                        HttpPost post = new HttpPost(url);
   StringEntity se = new StringEntity( json.toString());
   post.setEntity(se);
                        
   HttpResponse response = client.execute(post);
   return response;
 }
    public HttpResponse callGETService(String url) throws ClientProtocolException, IOException
    {   
                        // Call GET of REST Service 
                        DefaultHttpClient client = new DefaultHttpClient();
                        client = WebClientDevWrapper.wrapClient(client);

                        Credentials creds = new UsernamePasswordCredentials("username", "password");
                        client.getCredentialsProvider().setCredentials( new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT), creds);

                        HttpGet get = new HttpGet(url);
                        HttpResponse response = client.execute(get);
                
                        return response;
        }

    public static void main(String[] args) throws ClientProtocolException, IOException
    {
        RestService rs = new RestService();
 /*       String inputString="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + 
        "<root>\n" + 
        "   <description>Product New one</description>\n" + 
        "   <sku>146-4100-901</sku>\n" + 
        "</root>";
        
        // Create the product with SKU mentioned above.

        HttpResponse response = rs.callPOSTService(inputString,"https://abc.com/api/v1/products");
        // Get the URL to update the product - Eg: "https://abc.com/api/v1/products/18"
        Header header = response.getFirstHeader("Location");
        String update_url=header.toString().substring(10);
        System.out.println(update_url);*/

       /* // Update the product
        String updatedinputString="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" + 
        "<root>\n" + 
        "   <description>Update Product New one</description>\n" + 
        "   <family>Items</family>\n" + 
        "</root>";
        System.out.println(rs.callIPOSTService(updatedinputString,"https://abc.com/api/v1/products/22"));
        */
        // Get Product
        HttpResponse response = rs.callGETService("https://abc.com/api/v1/products/22");
        String xmlResponse = "";
        BufferedReader rd = new BufferedReader (new InputStreamReader(response.getEntity().getContent()));
        String line = "";
        while ((line = rd.readLine()) != null) {
          xmlResponse+=line;
        }
        System.out.println(xmlResponse);
    }

}

WebClientDevWrapper.java - Used to bypass the SSL

package oracle.soa.rest;

import java.io.IOException;

import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLException;
import javax.net.ssl.SSLSession;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.conn.ssl.X509HostnameVerifier;
import org.apache.http.impl.client.DefaultHttpClient;

/*
*/
public class WebClientDevWrapper {

    public static DefaultHttpClient wrapClient(DefaultHttpClient base) {
        try {
            SSLContext ctx = SSLContext.getInstance("TLS");
            X509TrustManager tm = new X509TrustManager() {

                public void checkClientTrusted(X509Certificate[] xcs,
                                               String string) throws CertificateException {
                }

                public void checkServerTrusted(X509Certificate[] xcs,
                                               String string) throws CertificateException {
                }

                public X509Certificate[] getAcceptedIssuers() {
                    return null;
                }
            };
            X509HostnameVerifier verifier = new X509HostnameVerifier() {

                @Override
                public void verify(String string,
                                   SSLSocket ssls) throws IOException {
                }

                @Override
                public void verify(String string,
                                   X509Certificate xc) throws SSLException {
                }

                @Override
                public void verify(String string, String[] strings,
                                   String[] strings1) throws SSLException {
                }

                @Override
                public boolean verify(String string, SSLSession ssls) {
                    return true;
                }
            };
            ctx.init(null, new TrustManager[] { tm }, null);
            SSLSocketFactory ssf = new SSLSocketFactory(ctx);
            ssf.setHostnameVerifier(verifier);
            ClientConnectionManager ccm = base.getConnectionManager();
            SchemeRegistry sr = ccm.getSchemeRegistry();
            sr.register(new Scheme("https", ssf, 443));
            return new DefaultHttpClient(ccm, base.getParams());
        } catch (Exception ex) {
            ex.printStackTrace();
            return null;
        }
    }
}

The following are the libraries added to the class path:



I googled and found all of these in the net and added to the class path.

The main() method provides a way to
1. perform GET to retrieve the records using the URL
2. perform POST to create.
3. perform POST to update.

The Java code can be called via a BPEL/Mediator component using Java Embedding ( or more cleaner way by using a Spring component). The code in main method could replace the most of the code in the Java embedding. Most importantly we need to create an XML schema containing the request/response in accordance to the JSON request/response schema of the REST URI.
Eg : In JSON

{
    "sku":"130-01-099",
    "description":"Test"
}

equivalent XML for the same

<?xml version="1.0" encoding="UTF-8" ?>
<root>
   <sku>130-01-099</sku>
   <description>Test</description>
</root>

You could use the following tool for this conversion - http://www.utilities-online.info/xmltojson/#.U9_NevldWAg
You can create an XSD based on the above XML and accordingly use it in the BPEL process to call the Java embedding code/Spring Context component that calls REST service using JSON messaging standard.

Conclusion:  

 While Oracle SOA Suite 12c provides an out of box solution for interacting with REST services with JSON data standard using HTTPAdapter, 11g still need to seek the help of Java code to achieve the same. The way of  creating a Java class that takes XML as input ,converting it to an equivalent JSON and  then calling REST service using JSON ,provides us an interoperability between SOA suite components those primarily still rely on XML messaging with REST services relying on JSON.


Wednesday, April 30, 2014

Understanding the need of Whole Server Migration in Oracle Fusion Middleware

Whole server migration (WSM) is the mechanism built into Oracle WebLogic server that makes it possible to move an entire server instance, along with all of the services it owns, to a different physical machine when triggered by a platform failure event. WSM is probably overkill for most service requirements, because in the vast majority of cases, services can be set up to run on all managed server instances in a cluster, and failover of services from one server to another is transparent and automatic without the need to migrate an entire server instance to another platform. A few critical services, however, with JMS and JTA transactions being the prime examples, have a different relationship with managed server clusters.  These services cannot be pulled out of a failed server instance and restarted in a different operational server instance; the server instance, IP address and all, has to be migrated together with these special services.  Such services normally fall under the purview of SOA managed servers.

Architects and the administrators of the HA design should be aware of the need of WSM based on the needs of the services that are being designed at the client. The following article tries to explain the need of the WSM for various service scenarios and needs. This would probably give some ( if not complete) fair idea on whether to consider setting up the Server migration or not. 

Ability to recover Instances using service engine:

 WSM is not needed ( unnecessary and costly) whenever the instances can be recovered by the service engine ( lets say BPEL). Recovery settings in EM can be set to automatically or manually recover instances at a scheduled interval or at system/server start. Hence Its important to know when an instance can / can not be recovered. In simple an instance can be recovered if it is dehydrated ( persisted in database) before it failed.
 Consider a asynchronous BPEL process 'A' with default transactional and onewaydeliverypolicy ie., "requiresNew" and "async.persist" set. Before the instance is created the message is persisted into the dlv_message table with in the parent transaction if one existed. Because this message is persisted into the database any failure / rollback of the instance would make it still possible to recover.


Ability to "Fail over":
Homogeneous services that are deployed cluster wide will be transparently be able to serve the requests from either of the two nodes in case any service on a node fails or goes down.
Consider a scenario where Service A calls Service B synchronously. Lets say based on round robin approach the Service B of Server 2 is given a chance to serve the request. While the instance is in progress if the service B fails for some reason ( service issues or server shutdown), the transaction is rolled back. Upon retrial it would be assigned the Server1's Service B and hence the transaction is smoothly fulfilled. This is called fail over. In case if the service B is asynchronous you may need to recover it manually from service engine console as explained above.
Server migration is not really needed for these services as they are homogeneous ( not heterogeneous or pinned to a particular server) and hence fail over with some disciplined retry and recovery methods would be sufficient.

Purpose of the Server Migration:
We do not need whole server migration if your service fall in either of above categories ( ability to recover or ability to fail over) However if your process is 

synchronous or Asynchronous  with oneWayDeliveryPolicy is set to 'sync' ( as below)
  <property name="bpel.config.oneWayDeliveryPolicy" type="xs:string"                   many="false">sync</property>
the bpel will participate in the same thread as the caller with out having the message persisted before an instance is created. In these situations a roll back of the instance would leave no trace of the message in the database and hence leaving no possibility to recover them using service engine. If your service is homogeneous a retry of the service ( by using retry option on the partner link or adapter) may kick fail over(ie., kick of the service from other node as explained in prev section). However if your service can not be failed over as the services are pinned to a specific server then the only option to recover them is to bring the server up in the same node or in the other node (server migration). Lets see this practically from an example.

Important Note : If the BPEL is
Asynchronous with oneWayDeliveryPolicy set to 'async.cache' ( as below)
  <property name="bpel.config.oneWayDeliveryPolicy" type="xs:string"                   many="false">async.cache</property>
the message is stored in in-memory cache and it will vanish upon server failure. As the message is not persisted there cant be any recovery to act upon. In this case the new thread and new transaction are just created , not persisted anywhere if the instance is roll backed for any reason , you have neither a chance to recover from BPEL engine nor using Whole server migration. That is because your thread and transaction are rolled back thus losing the message.So be careful when you design the service with async.cache.

Whole Server Migration Example : 
A sample Service Design that leaves no choice (to recover) other than Server migration.


Background :
  • WLS_SOA1 , WLS_SOA2 are two managed servers on machines SOAHOST1,SOAHOST2 clustered ( SOA_Cluster) in a weblogic domain with SOA installed and configured on them.
  • The transaction and jms logs / persistent stores are stored at common location accessible to both these machines.
  • Whole server migration is NOT setup.
  • DemoOutQueue,DemoInQueue are Uniform Distributed Queues. 
  • Service A is an asynchronous service with onewaydleiverypolicy set to 'sync'. It consumes messages placed on the DemoOutQueue and places it onto the DemoInQueue. A delay of approx 30 secs is introduced between consuming and producing using some for loops. It is deployed on the SOA Cluster.
  • Auto recovery on start up and for scheduled intervals are set to off.

Experiments :

Test 1:
Action 1 : 
Five custom messages ( compatible with the jms adapter design) with id's 1,2,3,4,5 were placed on to the DemoOutQueue.
Wait for 15 seconds and shutdown the WLS_SOA2 server.

Observation for Action 1:
Messages 1,3,5 are processed successfully by WLS_SOA1.
Messages 2,4 went to Non Recoverable state as the server WLS_SOA2 processing them was abruptly shutdown.
In few seconds after the WLS_SOA2 is shutdown, the messages 2.4 are picked up by the Service A of WLS_SOA1 and were processed successfully.
All messages 1,2,3,4,5 were placed in DemoInqueue.

Test 2: Repeat the TEST Case 1 again with different message id's for clarity and no confusion.

Action 1 : 
Five custom messages ( compatible with the jms adapter design) with id's 6,7,8,9,10 were placed on to the DemoOutQueue.
Wait for 15 seconds and shutdown the WLS_SOA2 server.
Observation for Action 1:
Messages 6,8,10 are processed successfully by WLS_SOA1.
Messages 7,9 went to Non Recoverable state as the server WLS_SOA2 processing them was abruptly shutdown.
Wait for some time to see if anything happens.
Waited but nothing else happened.
Only messages 6,8,10 were placed in DemoInqueue. 

Action 2: 
Start the WLS_SOA2 server.
Observation for Action 2: 
Messages 7  was picked up by the Service A on WLS_SOA1 and Message 9 was picked up by the Service A on WLS_SOA2 and were processed successfully by placing them on to the DemoInQueue.

Note : The results of Test 2 may be just like Test 1's shown above. You may need to run for a good number of times to get the result of Test 2 as shown above. vice versa..ie., The results of Test 1 above may be like Test 2 as shown above and you may need to run considerable amount of times to get what we got above. But if your web logic cluster is round-robin load balanced you are most probably going to see the results in just 2 runs.

Quick understanding of the UDQ behaviour in cluster before we jump on to the analysis:

Uniform distributed queue is a queue distributed across the different servers cluster wide ie., for an client (internal or external) the queue appears as one lets say DemoOutQueue. However internally the DemoOutQueue UDQ is/are two queues each targeted to JMS servers on WLS_SOA1 and WLS_SOA2,
lets say DemoOutQueue1 ,DemoOutQueue 2 respectively for simplicity.You can see this in Admin console -> JMS Modules - Distributed Queue -> Monitoring

When a message/ set of messages are placed on a UDQ ( here DemoOutQueue) , the messages are placed either on DemoOutQueue1 or DemoOutQueue2 but not on both for Data consistency. This is called the singleton or pinned behaviour of the JMS.

Analysis :

Test 1 - Observation 1 - Reasoning :  Messages 1,2,3,4,5 when placed on DemoOutQueue, the cluster load balancing chose DemoOutQueue1(ie., DemoOutQueue hosted on JMSServer of WLS_SOA1) and placed all the messages on DemoOutQueue1.
Service A of both WLS_SOA1 and WLS_SOA2 servers( lets call ServiceA1,ServiceA2 for simplicity) picked the messages in the round robin fashion ie., ServiceA1 picked 1,3,5 and ServiceA2 picked 2,4 and created instances respectively.When the WLS_SOA2 is shutdown, the Service A2 instances for messages 2,4 were rolled back ( as they were not yet finished with processing) and as the Service A is sync, the messages rolledback to the original queue from where they have come from. In this case it is DemoOutQueue1.However as still WLS_SOA1 is up and running the Service A on WLS_SOA1 ie., ServiceA1 picks these new messages 2,4 and processes them. Hence we see all the messages were successful.

Test 2 - Observations 1,2 - Reasoning : Messages 6,7,8,9,10 this time were placed on DemoOutQueue2 ( ie, DemoOutQueue of JMS Server targeted to WLS_SOA2). ServiceA1 picked 6,8,10 messages. Service A2 picked 7,9 messages and instances were created respectively. However as WLS_SOA2 was shutdown the instances being processed by ServiceA2 ie., for messaged 7 and 9 were abruptly rolledback to the DemoOutQueue2. But now as the WLS_SOA2 is shutdown any consumers of DemoOutQueue2 will not get any messages as DemoOutqueue2 is literally not there ( down because of its server shutdown) for giving the messages. Hence nothing happens.
When the Server WLS_SOA2 comes back the messages 7,9 are picked by ServiceA1,ServiceA2 respectively in round robin fashion and are processed successfully placing the messages on to the DemoInQueue.

Quick Summary : As you can now understand, if there are any messaged placed on to the queue that was targeted to a server that went down, those messaged are not available to be failed over. Only solution is to get the queue up. The only possible approach for this is either get the Server up or get it migrated in case of multiple unsuccessful retrials on a particular node. 

Conclusion : 
Whole server migration involves a tightly-orchestrated sequence of events with participation from WebLogic, Coherence, O/S, network, and database components gets triggered if a SOA server goes down in a Fusion Applications infrastructure. This is costly , time consuming and also needs an extensive amount of hardware sizing and capacity planning to host the migrated server. One needs to really see the possibility of making things work with out WSM by designing services to fall under recovery of service engines or fail over scenarios. However if you extensively need to use JMS distributed destinations and XA transactions spanning multiple resources ( multiple databases) on a two phase commit then WSM is the way to go.

Note : Service Migration is another possible approach but it is not yet supported with Oracle BPEL/OSB. Refer articles NOTE:1306533.1 and NOTE:1407715.1.For service level migration support on SOA, there is a enhancement requests (SOA 11g: BUG:13447082) raised and for OSB (OSB 11g: BUG:13446665) as well. There are still under development and will be included in future releases.

This is still only one use case explaining the need for the Whole Server Migration where JMS services are pinned to server. In my next blog I shall talk more about JTA recovery service use cases.

Disclaimer: The above experiments are based on my own observations and setup of the environment on my local machines. A possibility of a different behaviour may be possible in a real production infrastructure, although highly unlikely to differ from this.

References:
http://docs.oracle.com/cd/E14571_01/web.1111/e13709/migration.htm
http://www.ateam-oracle.com/floating-ip-addresses-and-whole-server-migration-in-fusion-applications/#comment-98
https://community.oracle.com/thread/3527201
Oracle EDG
https://blogs.oracle.com/soabpm/entry/soa_suite_11g_-_transactions_b_1

Wednesday, March 12, 2014

Questions one should have answers for while setting High Availability in Oracle SOA

The customers who are looking to embark on Oracle SOA for the first time especially on a high available mode are mostly daunted looking at various deployment topologies and administrative tasks involved with in. While the Oracle Enterprise Deployment Guide ( EDG) is very popular and is used as a reference to setup the high availability there would be still a lot of questions lingering around like why certain things are being setup or setup in a particular way. In fact, sometimes,its not clear why we are doing somethings in the entire process. This post attempts to help resolve those vague concepts.

Question : What is the use of external load balancer(software or hardware) when we plan to implement a web logic cluster that does load balancing anyway.?

Answer :  External Load balancers provide an additional layer of security and more reliability of load balancing as they are not affected by the internal systems performances but are used only to load balance across Http servers or some cases ( not suggested) application servers using well defined algortihms. The need for load balancing at cluster level is for clustered object stubs when no algorithm is specified at the object level. The algorithm at weblogic cluster level supports load balancing for RMI objects and EJBs.

Question : Can Managed servers continue to run in case admin server comes down ?
Answer : Yes of course they do. They try to contact admin server at a particular frequency that is setup int he configurations, but then will continue on their own. Another point to note is that managed servers can also start by themselves ( with help of node manager) with out Admin server being up. This is called as MSI mode or Managed Server Independence mode.The problem with this mode is that managed servers can not get the latest update on the configurations done in the domain as it runs with its own copy of the configuration which is collected upon its start when admin was up and running.

Question : What is whole server migration ? Why do I need it when my cluster is performing a "fail over" of various services when one of its members is down ?

Answer : In a WebLogic Server cluster, most services are deployed homogeneously on all server instances in the cluster, enabling transparent failover from one server to another. In contrast, "pinned services" such as JMS and the JTA transaction recovery system are targeted at individual server instances within a cluster—for these services, WebLogic Server supports failure recovery with migration, as opposed to failover.

Question : Why does JMS and JTA are targeted to single server ?

Answer : Weblogic services are divided into two categories , one Clustered Services and two Singleton Services. The JMS,JTA services come under the Singleton services that can be targeted only to a single instance in a cluster and more importantly will be managed by a single persistent store ( Note : single persistent store is the key).
The reason for this setup is that it offers high QOS by ensuring message consistency and reliability. For example a JMS message on a topic or a queue should not be stored in multiple places for data consistency. Also the information related to the message like transactions, acknowledgement etc should not be shared across.

Question : Is Whole Server migration the only way to ensure JMS and JTA "Singleton" services are recovered in case of failures?

Answer : No. We can do service migration too. ie., migration of individual JMS and JTA services to other server. The problem is that only JTA service migration supports the migration to other server using the default persistent store. Other services such as JMS needs to use custom persistent stores for them to be able to migrate to other servers.

Question : Would the problem be same for any other Singleton services in my server?
Answer : Yes. The singleton services as the name indicates, means that they are meant to be targeted to a single server instance and will use one persistent store allocated for those services in that server instance.

Question : What is the impact if Server/Service migration is not setup ?
Answer  :  Your instances that was running crashes and it may be retried based on various parameters. If its an asynchronous instance that is dehydrated by the time it crashed, the server may try to recover it. Manual recovery sometimes is still the option in certain cases.

This still sounds theoretical although it clears a cloud of vagueness regarding some important concepts of HA architecture. Understanding of the above in more practical way with examples will be done my upcoming blog.

Quick recap of JVM in context of weblogic

This is probably an age old concept that was discussed many times since the beginning of java but it seems like many programmers struggle or often lose track of these trivial but important basics.This post is to just refresh that knowledge. Specially the people who are working on weblogic server often encounter issues such as "OutOfMemoryError" , "PermGenSpace Out of memory" and end up tweaking these settings with no or less real understanding.

Lets see what different JVM settings are-( not full list)

1. Xms - Initial heap space.
2. Xmx - Maximum heap space.
3. XX (or) XX:PermSize  - Permenant Generation heap space.


Initial heap space - The amount of heap space allocated to the JVM heap at the time of server start.

Maximum heap space - Once server is started and applications/instances are deployed / redeployed , the available (initial) heap space is utilized( free space decreases) and also is fragmented to some extent.
Fragmented means that the heap memory is available as not one big chunk at one place but rather is scattered at various multiple places.The problem with the fragmentation is that in order to write / create a object in heap , the amount of memory at one location of heap space may not be sufficient to store the object entirely. As a result the object is created at two different places. This causes a same instance data / application data to be scattered across different parts in the heap making it very time consuming while reading.
Now lets come back to what maximum heap space is about. Because of rapid use of applications if the increase of heap space becomes equal to the initial heap space we allocated , the operating system allocates more heap space based on need upto the max heap space.

How do I set these two parameters for better perforamce ?
Answer : Depends on your need. Lets consider two cases a) Where you know that you are going to run huge load on the server that needs much heap space. b) Where you ddont know how heap space is needed initially. In case of former(first case), you should allocate initial heap pace = max heap space ( -Xms equals to Xmx), as this would avoid an unneccassary allocation of memory by operating system once the initial heap space threshhold is reached. But as in this case initial heap is set maximum high OS doesnt intevene in between and hence can save good time.But setting a high heap right from the beginning is also not good for applications that fall into the second category where the growth of instance beyond initial heap space is unsure.However,when you allocate more initial heap space, the server may take more time to start.

PermGen space - There are three kinds of racks( or Generations) with in the Java Heap.
a) Young Generation b) Old Generation c) Permanent generation( PermGen)
Young Generation - most recently created and running objects use this.
Old Generation - Old objects which are still live will be moved to Old generation heap space. They are not frequently used but are still live.
Perm Generation - The space allocated with in the java heap to store the classes and other permanent static files that needs to be always there in heap.The space allocation completely depends on the number of applications used( classes that you deploy) and on the way programming is done ( ie., usage of static functions).
Few other memory jargon -

Eden Space (heap - young generation):pool from which memory is INITIALLY allocated for most objects.
Survivor Space (heap - young generation):pool containing objects that have survived GC of eden space.
Tenured Generation (heap - old generation):pool containing objects that have existed for some time in the survivor space
Permanent Generation (non-heap - stack):holds all the reflective data of the virtual machine itself, stores class level details, loading and unloading classes (e.g. JSPs), methods, String pool
Code Cache (non-heap):HotSpot JVM also includes a "code cache" containing memory used for compilation and storage of native code.

In order to set custom memory arguments for each of the servers (Admin and/or Managed) , you need to set the USER_MEM_ARGS in each the respective servers env script ie., setSOADomainEnv.cmd / setOSBDomainEnv.cmd / setOEREnv.cmd respectively

set USER_MEM_ARGS=-Xms256m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=768m

However doing so poses a problem - 
setDomainEnv calls the respective Env' files of the managed servers in the same domain.
- lets say setSOADomainEnv.sh , setOSBDomainEnv.sh , setOERDomainEnvsh in the same order.
Now each of these env scripts have their own USER_MEM_ARGS set. As USER_MEM_ARGS is a common variable across it holds the value updated by the last(latest) script in the order. In this case for eg : setOERDomainEnv.sh. So now all server's get started with the same memory arguments which is very bad.

For the same , I have written a very small tweak / custom script inside the setDomainEnv.cmd/sh. This will read the server name being started and then sets the memory parameters accordingly

 @REM **********START CUSTOM SCRIPT**************  
 @REM This script is needed to set the right memory parameters to the JVM based on the server being started.  
 if "%SERVER_NAME%" == "AdminServer" (  
  set USER_MEM_ARGS=-Xms256m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=768m  
 )  
 if "%SERVER_NAME%" == "soa_server1" (  
  set USER_MEM_ARGS=-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=768m  
 )  
 if "%SERVER_NAME%" == "oer_server1" (  
  set USER_MEM_ARGS=-Xms256m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=768m  
 )  
 if "%SERVER_NAME%" == "osb_server1" (  
  set USER_MEM_ARGS=-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=768m  
 )  
 if "%SERVER_NAME%" == "bam_server1" (  
  set USER_MEM_ARGS=-Xms256m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=768m  
 )  
 if "%SERVER_NAME%" == "osr_server1" (  
  set USER_MEM_ARGS=-Xms256m -Xmx512m -XX:PermSize=256m -XX:MaxPermSize=768m  
 )  
 @REM Set the memory args in the same way above for any other servers apart from the above those participate in the same domain  
 @REM ********END CUSTOM SCRIPT*********************  

With this script you need not bother about setting the memory arguments in the individual scripts at all as this overwrites all of them.

Note :  This script has to be placed before the below lines in the setDomainEnv.bat

if NOT "%USER_MEM_ARGS%"=="" (
set MEM_ARGS=%USER_MEM_ARGS%
)

Update : I came across this blog by Antony Reynolds explaining the same issue for which he had a similar solution.

Garbage Collection:

The other option we can use to produce higher throughput is to garbage collection.
GC algorithms are of two types:
a) Parallel,Serial and b) Concurrent.
Parallel GC stops the execution of all the application and performs the full GC, this generally provides better throughput but also high latency using all the CPU resources during GC.Its mostly called as "Stop the world" GC where everything else stops while it runs.The reason it is named parallel because Multiple threads in parallel are allocated for GC. The only difference between this and serial GC is that in serial there is only single thread allocated for GC.

Concurrent GC on the other hand, produces low latency but also low throughput since it performs GC while application executes ( not in all phases of GC though).

The Hot spot JVM provides following options for GC
-XX:-UseParallelGC
-XX:-UseSerialGC

The JRockit JVM provides some useful command-line parameters for Garbage collection -

-XgcPrio:pausetime (To minimize latency, parallel GC)
-XgcPrio:throughput (To minimize throughput, concurrent GC )
-XgcPrio:deterministic (To guarantee maximum pause time, for real time systems)

Force Garbage collection:

I have found a great simple program over net that forces garbage collection. Below is the snippet of the below code that can be run
1. Go to your weblogic home and ie., eg: Oracle\Middleware\wlserver_10.3\common\bin
2. Run the command wlst forceGC.py.
The forceGC.py can be placed under the common\bin folder and its contents as below


 forceGC.py  
 # WLST script which calls GC.  
 from java.util import *  
 from javax.management import *  
 import javax.management.Attribute  
 print 'starting the script .... '  
 # please replace userid and password with your AdminServer userid and password  
 # plz change the IP adresss and port number accordingly  
 connect('weblogic','weblogic123',url='t3://localhost:7001')  
 state('AdminServer')  
 # For Force GC ....  
 domainRuntime()  
 cd('/ServerRuntimes/AdminServer/JVMRuntime/AdminServer')  
 print ' Performing Force GC...'  
 cmo.runGC()  
 disconnect()  
 print 'End of script ...'  
 exit()  


Imp Note : If you want to run the force gc for your managed server, then just change the state('AdminServer') to state('WLS_SOA1') where WLS_SOA1 is the managed server name. Make sure that Node managers are accessible and running.

Weblogic DMS spy a war deployed on weblogic admin server at url http://adminhost:port/dms/index.html provides us with very useful metrics.One of them is the JVM_Memory Set which gives us the used vs free heap and non heap memories as shown below






When I ran the forceGC.py, in few minutes my used heap memory came down. This is pretty useful in many development environment when your server is performing very slow because of less memory and not yet garbage collected.

Conclusion : Use JVM memory settings according to your application needs. Understanding of the basic jvm memory concepts help you to do better configuration and makes you troubleshoot more meaningfully.