Docker and Documentum. Part II

When I started exploring docker I pursued the following objectives:

  1. Find out what’s new in RHEL 7
  2. Try docker
  3. Try silent installation in Documentum

RHEL 7

Official list of changes could be found here. Below my own list:

  1. New anaconda GUI, now it looks like mobile app (I always was happy with TUI installer though):
  2. NetworkManager now has CLI (nmcli) and TUI (nmtui) interfaces, but NM still unusable in complex configurations, so I disabled it as I did before in RHEL 6
  3. GRUB2 replaced GRUB, now grub.cfg file looks like a program – bad move :
    #
    # DO NOT EDIT THIS FILE
    #
    # It is automatically generated by grub2-mkconfig using templates
    # from /etc/grub.d and settings from /etc/default/grub
    #
    
    ### BEGIN /etc/grub.d/00_header ###
    set pager=1
    
    if [ -s $prefix/grubenv ]; then
      load_env
    fi
    if [ "${next_entry}" ] ; then
       set default="${next_entry}"
       set next_entry=
       save_env next_entry
       set boot_once=true
    else
       set default="${saved_entry}"
    fi
    
    if [ x"${feature_menuentry_id}" = xy ]; then
      menuentry_id_option="--id"
    else
      menuentry_id_option=""
    fi
    
    export menuentry_id_option
    
    if [ "${prev_saved_entry}" ]; then
      set saved_entry="${prev_saved_entry}"
      save_env saved_entry
      set prev_saved_entry=
      save_env prev_saved_entry
      set boot_once=true
    fi
    
    ...
    
    
  4. systemd replaced upstart (the most controversial innovation) – at first glance I liked it!
  5. OOTB vmware tools. Here I faced with performance problem: after a couple hours of uptime VM performance terribly slow downs:
    but ESX does not diagnose any performance issues, moreover it display weird counters for CPU and Memory utilization: blacklisting vmw_balloon and vmw_vmci modules and disabling vmtoolsd service solve performance issue

Docker

After a short time of exploring Docker I realized that it is not designed to run multiple services in a single container. Why?

  1. Docker manages networking inside container is some weird way: it is not possible to assign a certain ip-address to specific container – Docker generates new ip-address for container upon every container start:
    ~]# docker inspect --format '{{ .NetworkSettings.IPAddress }}' 22db680586e2
    172.17.0.2
    ~]# docker stop 22db680586e2
    22db680586e2
    ~]# docker start 22db680586e2
    22db680586e2
    ~]# docker inspect --format '{{ .NetworkSettings.IPAddress }}' 22db680586e2
    172.17.0.22
    ~]# docker stop 22db680586e2
    22db680586e2
    ~]# docker start 22db680586e2
    22db680586e2
    ~]# docker inspect --format '{{ .NetworkSettings.IPAddress }}' 22db680586e2
    172.17.0.23
    

    Jérôme Petazzoni wrote a workaround that allows to assign specific ip-address to container, but I found that solution bit awkward and decided to write my own tool similar to pipework, but I’ll write about that in later posts

  2. Docker does not properly shutdown containers which execute /sbin/init as entrypoint. Here you can find a good description of a problem and related discussion. I thought that replacing standard init is not a good idea, so I wrote a simple workaround to shutdown such containers properly:
    #!/bin/sh
    
    CONTAINER=$1
    TIMEOUT=${2:-300}
    WAIT_INTERVAL=10
    
    if [ "x$CONTAINER" = "x" ]; then
      echo "Specify container id"
      exit 1
    fi
    
    INIT_PID=`docker inspect --format "{{ .State.Pid }}" $CONTAINER`
    STATUS=$?
    
    if [ "x$STATUS" != "x0" ]; then
      echo "Unable to inspect container $CONTAINER"
      exit $STATUS
    fi
    
    if [ "x$INIT_PID" = "x0" ]; then
      echo "Container $CONTAINER is stopped"
      exit
    fi
    
    nsenter -t $INIT_PID -m -u -i -n -p /bin/bash >/dev/null 2>&1 <<__EOF__
    export PATH=/bin:/sbin:/usr/bin:/usr/sbin
    shutdown -h now
    __EOF__
    
    let total_wait=0
    while [ "x$INIT_PID" != "x0" ] && [ $total_wait -lt $TIMEOUT ]; do
      sleep $WAIT_INTERVAL
      let "total_wait=$total_wait+$WAIT_INTERVAL"
      ps -o pid --no-headers --ppid $INIT_PID >/dev/null 2>&1
      STATUS=$?
      if [ "x$STATUS" != "x0" ]; then
        break
      fi
    done
    
    docker stop $CONTAINER
    
    

Silent installation and Documentum

When EMC released xCP 2 they brought together all non-popular products in Documentum product stack (like xPlore, CTS, BAM, BPS, CIS and thumbnail server) and made those products mandatory for xCP 2 deployment. To make customers much happier they released product called xMS, that (in theory) was designed for reliability and easy maintenance of resulting mess of products. In practice, I found that xMS is absolutely unreliable and does not solve any maintenance problems. The only useful thing I found in xMS is a set of recipes for silent installation of Documentum products, these recipes could be found in WEB-INF/lib/recipes directory of xMS web-archive. That was a good starting point, unfortunately those recipes were intended for Windows environment, so some decompilation of Documentum installer was required to understand how create silent installation in Linux environment.

Creating a golden image is a cornerstone of Docker infrastructure. What needs to be put in this image? Without doubt, the simplest way is: install all Documentum software (and dependencies), setup docbase, and then create filesystem image – in such way EMC distributes their “Development edition” of Content Server. But is it OK to run several development environments having the same docbase name/id? I don’t think so. That’s why I created an image with following preinstalled (but not configured!) products: Oracle Express Edition (both database and client), Documentum Content Server 7.1 (and latest patchset), Documentum Process Engine, also I included into image startup/shutdown scripts for Documentum products and bootstrap procedure purposed to configure installed products upon first container “boot”. All related scripts could be found here.

With designed scenarios fully automated creation of golden image takes about 12 minutes:

[root@rhel70docker01 install]# ls -1
Content_Server_7.1_linux64_oracle.tar
CS_7.1.0060.0200_linux_ora_P06.tar.gz
docstop.sh
documentum.sh
firstboot.sh
image.sh
image.sh.log
jbosscontrol.sh
oracle-instantclient11.2-basic-11.2.0.4.0-1.x86_64.rpm
oracle-instantclient11.2-devel-11.2.0.4.0-1.x86_64.rpm
oracle-instantclient11.2-sqlplus-11.2.0.4.0-1.x86_64.rpm
oracle-xe-11.2.0-1.0.x86_64.rpm
Process_Engine_linux.tar
startdjms.sh
start_documentum.sh
stopdjms.sh
stop_documentum.sh
tomcatcontrol.sh
[root@rhel70docker01 install]# export TMPDIR=/u01/tmp
[root@rhel70docker01 install]# hostname localhost
[root@localhost install]# time ./image.sh >image.sh.log 2>&1

real    11m40.249s
user    4m17.899s
sys     1m25.055s
[root@localhost install]# grep password image.sh.log

  ...

!! Shell password for user root: R08qHdqomKLu
!! Shell password for user dmadmin: hlUCf95YZ72O
!! JMS admin password: 9zmC15gzNEqA
[root@rhel70docker01 install]#

The bootstrap procedure is designed to be run interactively through ssh:

[root@bfdd4d148706 ~]# /etc/init.d/firstboot start
Starting firstboot:
Enter repositroy name: docker_test
Enter repositroy id: 140801
Generating DATABASE_ADMIN_PASSWORD
Generating REPOSITORY_DB_PASSWORD
Generating REGISTRY_PASSWORD
Initializing oracle database...

or non-interactively through ssh:

~]# ssh root@172.17.0.3 \
> "REPOSITORY_NAME=docker_test REPOSITORY_ID=140801 /etc/init.d/firstboot start"
root@172.17.0.3's password:
Starting firstboot:
Generating DATABASE_ADMIN_PASSWORD
Generating REPOSITORY_DB_PASSWORD
Generating REGISTRY_PASSWORD
Initializing oracle database...
Initializing Documentum docbase...

or through docker “run” command:

~]# docker run -d -t -e REPOSITORY_NAME=docker_test -e REPOSITORY_ID=140801 documentum:71
6f9ee949b96e1117cffbd12c17e5573e139b5f224d2c2e2c3327a4bea474cca4

Now I can get working Documentum environment in 40 minutes:

[root@localhost install]# time docker run -t -d documentum:71
7c9e7b7b254bebe4a68f9af32ecaa44f72b18725b48a0338f91f1729351937be

real    0m0.752s
user    0m0.012s
sys     0m0.009s
[root@localhost install]# docker inspect \
> --format '{{ .NetworkSettings.IPAddress }}' 7c9e7b7b254b
172.17.0.5
[root@localhost install]# time ssh root@172.17.0.5 \
> "REPOSITORY_NAME=docker_test REPOSITORY_ID=140801 /etc/init.d/firstboot start"
The authenticity of host '172.17.0.5 (172.17.0.5)' can't be established.
RSA key fingerprint is e8:44:95:b2:3e:38:a2:9b:14:b5:6e:75:f2:97:b2:bd.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.5' (RSA) to the list of known hosts.
root@172.17.0.5's password:
Starting firstboot:
Generating DATABASE_ADMIN_PASSWORD
Generating REPOSITORY_DB_PASSWORD
Generating REGISTRY_PASSWORD
Initializing oracle database...
Initializing Documentum docbase...
Preparing to install...
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...

Launching installer...

Stopping documentum: [  OK  ]
Shutting down Oracle Database 11g Express Edition instance.
Stopping Oracle Net Listener.

Starting Oracle Net Listener.
Starting Oracle Database 11g Express Edition instance.

Starting documentum: [  OK  ]

real    37m36.877s
user    0m0.243s
sys     0m0.147s

3 thoughts on “Docker and Documentum. Part II

  1. Pingback: Dockerized xCP Dev Environment | Documentum Things
  2. Pingback: Beware of Thumbnail Server | Documentum in a (nuts)HELL
  3. Pingback: Docker and Documentum. Part IV | Documentum in a (nuts)HELL

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s