OpenSSL and the certification chain verification – CentOS / Linux using s_client

If one needs to validate / verify the certificate chain and the following message is spewed out by “OpenSSL” on the client (s_client or any tool that utilized OpenSSL under the hood such as curl or wget)  then this implies that either the server is really setup for SSL / TLS using a self-signed  certificate or the client does not have access to the root to validate the server certificate that is being returned by the SSL server as part of the “Server Hello” step of the SSL handshake. [One can utilize the " -showcerts" option to delineate and confirm either of the two situations elucidated above]

What we really need to see as a result of a successful verification of the server certificate is:

In order to achieve this with s_client, we can either utilize the "CApath" option or the "CAfile" option. [man s_client for more details]

The two options to achieve successful verification of the server certificate (Option 2 is a little more straightforward and you can try that first):

Option 1: [check to see the default location of the version of OpenSSL that is being used]

$ openssl version -a


OpenSSL 1.0.1c 10 May 2012
built on: Tue Nov 13 04:53:59 UTC 2012
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(idx,cisc,16,int) idea(int) blowfish(idx)
OPENSSLDIR: "/usr/local/ssl"

You are interested in the OPENSSLDIR setting where you would find the "certs" folder and that is the default directory of trusted certificates. The certificate of the root Certification Authority (CA) should be a constituent of this folder but there is a peculiarity which is that the certificate should be named or symbolically linked as in "hash.0" where "hash" is the hashed certificate subject name. One could use "c_rehash" script to automate this part.

Copy the root CA certificate to the "certs" folder. And:


$ c_rehash certs [this would do the trick]

$ ll certs [Check the folder to make sure that the necessary symbolic link is created]
total 4
lrwxrwxrwx 1 root root   10 Nov 16 07:34 beebff74.0 -> cacert.pem
-rw-r--r-- 1 root root 1135 Nov 16 07:26 cacert.pem

Run s_client:

$ openssl s_client -connect myHost.com443  -CApath /usr/local/ssl/certs/

And voila, the certification validation is successful since at the end we have:

Verify return code: 0 (ok)

Option 2: [Use the CAfile option]

$ openssl s_client -connect  -CAfile /app/sslca/cacert.pem

That should lead to the same result of successful server certificate validation.


Copy (scp without password) files to Amazon EC2 instance

While copying files (scp) to Amazon EC2 instance, got the following error:

$ scp ~/Downloads/pcre-8.31.tar.gz  -i /Users/khanna/Downloads/k2.pem
Permission denied (publickey).
lost connection

The error message is not that intuitive to debug but after some researching it turned out that the order of arguments was incorrect. This fixes that:

$ scp  -i /Users/khanna/Downloads/k2.pem  ~/Downloads/pcre-8.31.tar.gz



I was interested in figuring out the time taken for a HTTP SSL handshake and the following file download time and after a discussion with colleagues decided in utilizing ssldump.


The ssldump utility can either read the output of tcpdump or could simply be set to listen on a particular interface as well. Some examples follow:

Listen on the Loopback interface  “lo”:

Note that if the client and server are on the same machine then this is what you might need to do.

Listen on a particular interface:

It could also read the generated output of tcpdump:

However if you have network traffic captured as a result of running tcpdump and running ssldump on it leads to “ERRORLength mismatch” message, then one needs to increase the packet capture size. This is what I did for a quick check:

Here the value “0” is assocated with the “snarflen” (-s) argument and it implies that we “use the required length to catch whole packets” (from the man page of tcpdump).


For ssldump troubleshooting, please refer to:

Python 2.7.3 install on Linux

To use a later version of Python to what the RPM is available for, one needs to build and install it.

This is what I did and it was a breeze to get that to up:

And that is it. If you run into issues, one can always start from scratch by running:

and then continue with the 3 steps that were outlined earlier viz.

How to get to the memory profile of your linux system – free, top, vmstat (free versus top versus vmstat)


There are a myriad of tools and scripts that one runs on Linux to figure out the important question of how much free memory is left viz. “how much RAM is available?”.
And to answer the query, we use the following 3 tools that are almost guaranteed to be on all Linux systems and some Unix variants:
  1. free
  2. top
  3. vmstat


The following command line demonstrates the invocation of the “free” command to display the amount of free and used physical and swap memory in the system in megabytes – borrowing the description from the man pages.


Free memory => free + buffers + cached = 155 + 314 + 1171 = 1640 =~ 1641 [the value in 3rd row. 4th col]



Free memory => 159100k + 321752k + 1199916k = 1680768k =~ 1641M (this is the value that we arrived at from the “free” coammand above]


Using vmstat with the ‘s’ switch to display the memory statistics, we have the following:


Free memory => free memory + buffer memory + swap cache = 157704 + 322028 + 1201220 = 1680952 = 1641.554M =~ 1640M

The paths to discovering memory statistics are many but they ultimately lead to the same figures.  :-)

Apache web server httpd 2.4.3 Build / Compile from source and ECC (Elliptic Curve) support (RHEL or CentOS Linux)


An earlier post had detailed setting up Apache httpd 2.2.22 with both DSA and RSA support in terms of SSL/TLS authentication. This post will detail setting up Apache httpd 2.4.3 with support for all three ciphers viz: RSA, DSA and ECC. The earlier post also covered the OpenSSL 1.0.x installation that supports all of these ciphers.

Please note that Apache httpd version 2.2.x does not have ECC support built in and it needs to be patched for ECC. However support for ECC is in trunk for the 2.4.x branch and that is the path that we will take.

Building Apache

  1. Download the source.
  2. Build./configure --prefix=/app/install/myinstalls/httpd-2.4 --enable-mods-shared="all ssl deflate disk-cache expires headers info cache proxy proxy-ajp proxy-balancer proxy-connect proxy-ftp proxy-http rewrite" --with-ssl=/usr/local/ssl --with-included-apr --with-pcre=/usr/local

    Note that we are utilizing the provided APR (Apache Portable Runtime) and are also pointing to the PCRE deployment. Please see the Prerequisite section below on the reasons for this.If there are any issues, run the following before retrying:




Apache Portable Runtime (APR) and utils might need to be updated or installed if the following error is printed on the screen while configuring Apache httpd which is the first step in the build process. If while running configure, the following is spewed out then you need to download and install APR and APR-UTIL.
configure: error: APR not found. Please read the documentation


Steps for APR (1.4.6) and APR-UTIL (1.4.1)  setup:

  1. Download the source into “[Apache HTTPD build location]/srclib”. Extract it and make sure there are no version numbers in the folders.
    From the Apache httpd documentation ( the latest versions of both APR and APR-Util from Apache APR, unpack them into ./srclib/apr and ./srclib/apr-util (be sure the domain names do not have version numbers; for example, the APR distribution must be under ./srclib/apr/)
  2. Append the following to the Apache httpd configure command:
  3. Continue with the Apache httpd configure process.


If during the Apache httpd build process, the following is spewed out then we need to build and install PCRE.
configure: error: pcre-config for libpcre not found. PCRE is required and available from


Steps for PCRE (8.31) setup:

  1. Download the PCRE source. Save it at any location.
  2. The PCRE build and install process will generate both shared and static libraries and that implies we do not have to explicitly require the dynamic libraries to be built.
  3. The configure command and the build process then is:
    ./configure --prefix=/usr/local
    make install
    If there is this error during the process then it implies that either libtools or GCC C++ compiler is not available:
    make[1]: Entering directory /app/install/myinstalls/pcre/pcre-8.31'CXX pcrecpp.lolibtool: compile: unrecognized option -DHAVE_CONFIG_H'libtool: compile: Try libtool --help' for more information.make[1]: *** [pcrecpp.lo] Error 1make[1]: Leaving directory /app/install/myinstalls/pcre/pcre-8.31'

    make: *** [all] Error 2

    Consequently, you would need to perform the following installs:


Errors related to mod_deflate and zlib

If the following error is spewed during configure:

checking for zlib location... not found


checking whether to enable mod_deflate... configure: error: mod_deflate has been requested but can not be built due to prerequisite failures

Then this implies that zlib or zlib-devel packages are missing or might need to be forced to be reinstalled. This should take care of installing them:

yum install zlib


yum install zlib-devel

Apache httpd 2.2.22 (or 2.4.x) and OpenSSL 1.x.x (RHEL or CentOS Linux) – build / compile / install Apache Web Server and OpenSSL with ECC (Elliptic Curve Cryptography) Accelarator on Linux (CentOS Red Hat)

The stack:

  1. Apache httpd (web server) version 2.2.22
  2. OpenSSL 1.x.x. (1.0.1c)
  3. RedHat Enterprise Linux 5.8 (rhel), CentOS or any flavor of Linux.

The goal:

To upgrade the OpenSSL library from 0.8.x to 1.x.x. The reason for the upgrade is the DSA algorithm support required in terms of installing a server certificate that is signed through the use of DSA 2048_256 CA key. Also to build and provide Elliptic Curve Cryptography (ECC) support with optimizations.


These are the problems or issues that had to be resolved in order to achieve the goal:

  1. OpenSSL 1.x.x. rpm not available. Therefore we needed to download the source and build it. And then it had to be installed in a location as not to overwrite the existing OpenSSL installation. Please note: the RedHat and CentOS linux (not to mention the other variants of linux) have a huge number of packages (I counted 500) that have dependencies directly or indirectly to OpenSSL. Consequently it is easier not to overwrite the OpenSSL installation with the later version; or remove the older version and install the later version.In a production system, you might want to do an “upgrade” but that is beyond the scope of this document.
  2. Apache 2.2.22 rpm not available (at least in the repository that we access). Consequently we would be downloading the source and building it as well.

Build and install process

This section details the effort and steps undertaken to install both OpenSSL and Apache httpd. We being with OpenSSL, install that and move on to Apache httpd.


Download the source and save it. Thereafter you have two options:

  1. Do you want to httpd to link statically to the generated static OpenSSL libararies (.a extension)?
  2. or dynamically to a shared library (.so)?

Although I found that the approach in bullet 1 led to the goal being easier to accomplish as there were issues in Apache httpd able to load the correct OpenSSL version with approach 2. This resulted in a lot of debugging to get it to work (the issue turned out to be the LDAP support that I was compiling with). However, in this document I will detail the second approach.

At the shell:

Now we check the installation to confirm the presence of shared libraries (.so files).

The creation of “” and “” is confirmed.

Optional Support for 64-bit optimized implementations of EC (Elliptic Curves)

To add support for 64-bit optimized implementations for NIST-P224, NIST-P256, NIST-P521, provide “enable-ec_nistp_64_gcc_128″ on the “configure” command line.


Apache httpd

We set the LD_LIBRARY_PATH to “/usr/local/ssl/lib” so as to make sure that the Apache httpd installation picks up the right libraries viz the right version – the version that has been installed as a result of following the steps in the preceding section.

export LD_LIBRARY_PATH=/usr/local/ssl/lib

We are assuming that a prior Apache Web Server version is not installed on the system. If it is then you could easily remove it if it is package managed.

To install Apache httpd, the following steps are to be followed (the proceeding texts details the arguments to configure):

If there are any errors or need to change the options to configure after having run it once, “make clean” and “make distclean” need to be run as well preceeding running of configure with the latest options.

If you need to statically link OpenSSL with Apache httpd, then one can configure httpd with the following (the following is a configure command to build Apache httpd with SSL support through the the mod_ssl module):


./configure --prefix=/app/install/myinstalls/httpd --enable-ssl --enable-rewrite --with-ssl=/usr/local/ssl


However, we would be dynamically linking to OpenSSL and therefore the configure command would be:


./configure --prefix=/app/install/myinstalls/httpd --enable-mods-shared="all ssl deflate disk-cache expires headers info cache proxy proxy-ajp proxy-balancer proxy-connect proxy-ftp proxy-http rewrite" --with-ssl=/usr/local/ssl


Please note that I have added support for a whole lot of modules as well. Another interesting piece to note is that ldap support is missing as is not the case with the following configure and options:


./configure --prefix=/app/installs/httpd --enable-mods-shared="all ssl authn-dbm authz-dbm auth-digest deflate disk-cache expires headers info isapi ldap cache proxy proxy-ajp proxy-balancer proxy-connect proxy-ftp proxy-http rewrite unique-id usertrack vhost-alias authn_alias mem_cache file_cache authnz_ldap charset_lite dav_lock disk_cache" --with-ldap


The reason for the missing ldap support is that having built Apache httpd for ldap support as in the above, the error_log has an entry pointing to the earlier version of OpenSSL and not the one that we compiled and installed as in the previous section:


[Sat Sep 01 00:48:16 2012] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/[EARLIER VERSION] DAV/2 configured -- resuming normal operations


Consequently, to enable the right version to be printed, we had to forsake LDAPsupport and the right version of OpenSSL was associated with the mod_ssl module as in:


[Sat Sep 01 00:48:16 2012] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/1.0.1c DAV/2 configured -- resuming normal operations


So in synopsis, these are the steps to build Apache httpd and dynamically link it with OpenSSL 1.x.x:


./configure --prefix=/app/install/myinstalls/httpd --enable-mods-shared="all ssl deflate disk-cache expires headers info cache proxy proxy-ajp proxy-balancer proxy-connect proxy-ftp proxy-http rewrite" --with-ssl=/usr/local/ssl


make install


If as a result of performing the steps delineated above, an error of this type is spewed:


:/usr/bin/ld: /usr/local/ssl/lib/libssl.a(s2_srvr.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with fPIC


The implication is that OpenSSL was not compiled to generate shared libraries. Please see the OpenSSL section above for details on how to achieve that.

Thereafter one could test that the mod_ssl module is linking to the correct OpenSSL libraries through the use of the “ldd” command:

Some of the alphanumeric details enclosed in the parenthesis are replace with “…” for brevity.



To generate all N size strings (of 0s and 1s)

Goal: To generate N size strings made up of two constituents:

  • 0 and
  • 1

This program is a rudimentary means to illustrate backtracking and recursion.

If we have N=2 then these are the N size strings:

  1. 00
  2. 01
  3. 10
  4. 11

And the code in Java is provided here:

Tomcat startup issues

If the preceding error message is spewed up while starting tomcat then the resolution is to make sure of these three things:

  1. the scripts in the bin folder have execute permissions set.
  2. The CATALINA_HOME environment points to the right tomcat installation
  3. The JAVA_HOME is properly set.

Oracle date and Unix time

Unix time: It is the number of seconds that have elapsed since Jan 01, 1970 12:00 AM   UTC (Coordinated Universal Time). The Jan 01, 1970 12:00 AM is termed as the Epoch. Consequently the Epoch is translated to Unix time 0. Note that you would find “Unix time” and “Epoch time” to be used interchangeably in documentation and usage. Also note that Unix time does not take into account leap seconds. Lets explain this a little as we go along.

UTC: It is the primary time standard used across the world. Every year or two, a second is added to the UTC to allow for vagaries in the earth’s rotation to keep it withing 0.9 seconds of UT1 (Universal Time that is based on the earth’s rotation). UT is equivalent to GMT. All timezones are defined in terms of UTC.

The Java System.currentTimeInMillis() returns the Unix time in the granularity of milliseconds. Please note that all of this is dependent on the sensitivity of the underlying OS – whether it stores time in the granularity of milliseconds or not.

In Oracle, date is granular to seconds and therefore maps appropriately to the Unix time.

To get the current date in Oracle down to seconds:

To get number of days (in Unix time => in seconds) time since the Epoch:

Here, 2440588 => Julian day of 1/1/1970 (starts at 12 midnight or 12 AM or 00h) where Julian day is the number of days since January 1, 4712 BC (12 noon). So this implies the number of days in Epoch time multiplied to result in the number of seconds.

To get current time represented as Unix time:

You could always “round” the above to get rid of the decimal part:

select ROUND((SYSDATE-TO_DATE(‘19700101′, ‘yyyymmdd’))*86400) from dual;

If you truncate as above then you could use the following query to get more accurate results. Another way to do the above is to use this query:

This sums up the number of days since the Epoch in seconds and the number of seconds since midnight to return the current Unix time.