TCP and the Address Already in Use

I have posted a few pages on "TCP and the Address Already in Use" issue.

The article explains the "Address Already in Use" issues that surface while load testing java applications that are communicate through the Java networking libraries.

It includes an overview of the TCP workings in context of this discussion and provides tips on resolving the issue at various stages of development. Please take a look and your suggestions would be incorporated.

The location of the article is on my website.

Oracle RAC and HACMP

Oracle Fault Tolerance

The successor to Oracle Parallel Server is the RAC (Real Application Cluster). In a RAC setup multiple instances access a single database. This implies that if an instance crashes, there are other instances that a connection can fail over to. This includes transactional fail over as well wherein a transaction can be retried on another instance when the one it was previously executing on has crashed.
Note: if the disk that the database is mounted on crashes then RAC will be useful as well.
In an active-active setup, all instances are active and thus the benefits include:

  • Fault tolerance through fail over
  • Scalability through load balancing

HACMP (High Availability Cluster Multi-Processing) versus RAC

HACMP is a clusterware that can be used with Oracle RAC as well. Either Oracle Clusterware or HACMP can be used to provide these services bulleted above. If there is more than Oracle (other apps such as NFS server etc) that has to be HA then use HACMP as the clusterware.

HACMP and Oralce Clusterware are complimentary

Note: SMP => Symmetric MultiProcessing.

Thread Pooling and Executors in Java

Thread Pooling in Java 5

Thread Pooling in Java 5 is configured using one of the implementations of the executor interface.
One implementation is ThreadPoolExecutor. This implementation, as the name, suggests is configurable with the no of threads in the pool, the maximum pool size, a rejection handler, a thread factory, a work queue.

Executor <———- ExecutorService <——– AbstractExecutorService <———– ThreadPoolExecutor <———- ScheduledThreadPoolExecutor

The significance of these configurable parameters is listed below barring the self explanatory parameters:

  1. Rejection Handler / Saturation Policies: This is an implementation that determines the policy to be used in cases when a task is to be rejected. The rejection could be because of many reasons: the work queue is filled up, the executor is shutting down etc. There are predefined policies as well that are detailed in the JavaDocs.
  2. Thread Factory: An implementation of this will be used by the executor (ThreadPoolExecutor in this case) to create the pool thread. This is very useful since a customized thread factory implementation can be setup to perform a variety of customizations. Such as: specifying an UncaughtExceptionHandler on a per thread basis.
  3. Work Queue: This is where the tasks that are to be executed are held at. One could configure the work queue to be of a certain size. The pool threads will pop off this work queue and execute the tasks based on a certain policy that in turn is based on the BlockingQueue implementation. For instance: if the work queue is a PriorityQueue then tasks will be popped off based on priority.

A ScheheduledThreadPoolExecutor can additionally be setup to execute tasks periodically. A task can be scheduled to execute after a given delay and at a specified interval thereafter. If a task takes longer than the interval specified to execute then another thread is spawned or reused from the pool to execute the task. This is different from a Timer in terms of the task always is executed after the interval even if the preceding task is still running.

Difference between executor obtained from Executors and directly instantiating the ThreadPoolExecutor or ScheduledThreadPoolExecutor

The distinguishing facet is that the Executor obtained from the new****Thread*** methods in Executors is preconfigured and cannot be reconfigured while it is running. Of course one can cast it to ThreadPoolExecutor (or any other implementation) and then invoke it’s configuration API but it is discouraged since that would require coupling the caller and the callee.
An Executor so obtained is preconfigured in terms of:
. Saturation Policies
. Thread Factory
. Work Queue

In order to customize, one would need to instantiate the appropriate Executor such as ThreadPoolExecutor or ScheduledThreadPoolExecutor.

Throttling of Producers:

When the consumer is slower than the producer then there has to be a manner in which to regulate the producer to generate fewer messages for the consumer. A "Saturation Policy" for an Executor can be setup that will deal with messages that can no longer be stored in the work queue (it might be capacity constrained). The existing policies either throw a RejectedExecutionException or drop the message or makes the calling thread run the task.
The "CallerRunsPolicy" can be used to throttle the producer if the caller thread is the producer. But in cases when that is not so, one would be required to code for throttling behavior.
One way to throttle would be:
    semaphore.acquire();
    try {
        exec.execute ( new Runnable() {
 
            pubic void run() {
                try {
                    ….. // code the task here
                    …..
                }
                finally {
                    semaphore.release();
                }
            }
        }
    }
    catch (RejectedExecutionException e) {
        semaphore.release();
    }
    

Task Cancellation Policies and Thread Interruption Policies

These two policies are related to each other. A well behaved task should have a clearly documented cancellation policy. Similarly a thread should have a clearly documented interruption policy. There are a myriad of reasons to implement a cancellation policy – the application needs to be shut down or the task is taking too long to run – being two of them. How does a task/thread respond to cancellation, how fast does it respond, what is the status of the task after it has been canceled are some of the questions that need to be answered.

These polices need to be well thought about and are of paramount importance when the application needs to be brought down and desired responsiveness and proper house cleaning are required. It so happens that the none of the Executor shutdown methods (shutdown() and shutdownNow() ) provide a list of tasks that have not been started yet.

A task once submitted to an Executor has 3 states:
1. Submitted
2. Started
3. Completed

Significance of Executor being Terminated:

An executor is marked for termination when the application needs to exit. A SIGTERM (kill) on UNIX will initiate the termination process. If proper and graceful shutdown of the application is required then the Executors, if any, need to be terminated as well.
To achieve that, the following API is invoked:
1. shutdown(): this executes all the queued tasks and in a case wherein there are 1000s’ of them might take undesired period of time to terminate thereby delaying the termination.
2. shutdownNow(): All tasks are started are cancelled (they might be an intermediate state) and tasks that are submitted and not started are returned.

As one can see, both the approaches above suffer from disadvantages.

My recommendation is to not use shutdownNow() and to use shutdown() and to code the tasks in a way that each checks for the status of the executor as soon as it is started and returning if a shutdown has been requested. In order to capture all such tasks that were in the work queue and started by the executor, the tasks should update a list. This list would be the list of tasks that were started and cancelled and are to be run again.

Note that one does not need to go though all these hoops if the tasks execution is idempotent.