EMC have shared CS 7.3

Have no idea what does mean IA acronym (internet access, internal affairs?) however:

  • AIX/Oracle got resurrected – seems that money triumph over evilstupidity
  • JMS is now on wildfly 9.0.1
  • bundled JVM version is 1.8.0_66
  • Linux/PostgreSQL is corrupted – try _old directory
  • composer.jar from Linux/Oracle is corrupted, borrow it from Linux/PostgreSQL
  • docker images seem to be corrupted too:
    docker load < centpgrccs 
    6941bfcbbfca: Loading layer [==================================================>] 1.024 kB/1.024 kB
    41459f052977: Loading layer [==================================================>] 224.7 MB/224.7 MB
    open /var/lib/docker/tmp/docker-import-813028725/dd1a6e7bc38e7d74bc5bb7be41e1d1d4f8d17d04cacef4cf93e3cc1ed16ce39e/json: no such file or directory
    
    
    [root@rhel72docker01 ~]# docker load < centpgseamlesscs 
    open /var/lib/docker/tmp/docker-import-785439904/be8abba669364a4895c80b2e4c06d8b391faffc54a190abc6aa91ed13d423140/json: no such file or directory
    
  • UPD

    By tradition I forgot to read installation guide before performing installation (actually it would be just a waste of time), in order to install Linux/PostgreSQL you need following:

    • PostgreSQL database
    • /etc/odbcinst.ini
    • /etc/odbc.ini
    • connection_string environment variable pointing to ODBC’s name
    • POSTGRESQL_HOME environment variable pointing to installation directory of PostgreSQL client libraries (for example, if psql is located in /usr/bin/psql POSTGRESQL_HOME should be /usr)
    • remove $DM_HOME/bin/liblber-2.4.so.2

    UPD2

    CS distribution media contains special java utility (MigrationUtil, sources are available) which allows to change hostname, installation owner and even docbase identifier and name of already installed repository, it seems that EMC considers idea about distributing CS as ready to run docker image seriously.

    UPD3

    MigrationUtil mentioned previously have turn out to be a piece of dog crap – you may use it only if you want to corrupt your database.

    UPD4

    Have no idea how is it possible to do docker without understanding it’s basic concepts – docker images are tar archives, not zip:

    Think soberly. Is it hard?

    Well, Momentum ended, TSGRP guys posted theirs thoughts about conference, now it is a time to ruin the myths.

    It is worth to say that I have never attended such eventsMomentum (and most probably will never do, no matter what company will held it in the future), because from my perspective such events should look like:

    EMC thinks that Momentum looks like:

    but in real life it looks like:

    Docker myth

    Issue #1 to address – Forced re-indexing and database changes are time consuming

    Patrick offered a stateless Documentum –

    • Point and run Content Server
    • Binary upgrades
    • No forced changes to database
    • Eliminates downtime from reindexing

    Able to bring in Bedrock server without any changes to database or repository.

    Makes upgrades easier (and faster). Has the ability to just run “Bedrock” against existing database and content repository due to everything due to Documentum in a Docker Container.

    • Portable Production Ready Images
    • Continuous delivery of upgrades and patches to production
    • Faster Rollback

    Documentum Application Containers

    • Developer – All in One container for Functional Investigation
    • Service Images – Individual services containers for Agile Expansion and Load balancing
    • Stateless Documentum – External Content, Database & Isolated Configuration for Fast Patching and Upgrades

    Different Containers for Content Store could be more containers for xCP or other customizations.
    Docker is Linux based (no support for Microsoft right now).

    How to migrate to Docker environment.

    • Add content server – Docker container – can operate alongside existing Documentum
    • Deploy extra Docker containers to manage high volumes
    • Migrate to Docker by turning off original non-Docker instances

    How many people are looking at Docker (only 1 out of 60 in room)
    Hot upgrades to D7.3+ (Bedrock – Feldspar…)

    • Can run Bedrock on Docbase 7.2 and also have it work on a Docbase Bedrock.
    • Run new version along side old version in Production

    Actually, everything that was “written” about Docker sounds ridiculous… The main problem is: Docker complicates everything. Why? Let’s elaborate. First of all, what is a Docker image? It is just a tar archive where maintainer put service’s binaries among with dependencies and bootstrap procedures: take a look at my shell scenario which creates document image – there is nothing related to Docker except the command which uploads archive into the Docker repository, there is no rocket science: I may unpack my archive into specific directory and run my services inside chroot take a look at chroot usecases – they are pretty the same as Docker’s:

    • Testing and development
    • Dependency control
    • Compatibility
    • Recovery
    • Privilege separation

    If chroot is so cool, what are the Docker’s advantages over chroot? I do believe that the answer for this question is obvious:

    But if you have chosen to use Docker you must understand that your deployment/maintenance workflow changes dramatically, for example, imagine that you decided to install a patch/update in Dockerized environment (no matter Documentum (i.e. application) or operating system (i.e. platform) patch), what steps do you need to perform to achieve your goal? These steps are:

    • update base image
    • test new base image
    • stop existing container
    • remove existing container
    • deploy new container

    This procedure, obviously, is less straightforward than installing patches/updates using puppet, ansible or something else or even manually, however, this is how Docker workflow designed and supposed to work. Now ask yourself how many typical Documentum installation do you need to have to start taking advantage of using Docker? In my opinion dozen is not enough, if you have more you are either a large enterprise or a big consulting company, and it is not a EMC business to lecture you how to manage your environments.

    Why you need dfc.session.keepalive.enable

    Have spent two days for troubleshooting another one DFC issue, however it was worth it.

    Two days ago my colleagues started complaining about weird Webtop performance – sometimes it’s response time was good, sometimes it stucks (actually, for their configuration, i.e. 48 Weblogic nodes, it is some kind of “expected” behaviour – the challenge is to find failure nodes). Further investigation revealed that problem nodes got stuck with following stack:

    "[STUCK] ExecuteThread: '114' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f3624096000 nid=0x7606 runnable [0x00007f34ae6e3000]
       java.lang.Thread.State: RUNNABLE
            at sun.nio.ch.FileDispatcher.read0(Native Method)
            at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
            at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
            at sun.nio.ch.IOUtil.read(IOUtil.java:171)
            at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
            - locked <0x000000061f780c10> (a java.lang.Object)
            at com.documentum.fc.impl.util.io.MessageChannel.readSocket(MessageChannel.java:123)
            at com.documentum.fc.impl.util.io.MessageChannel.readLength(MessageChannel.java:100)
            at com.documentum.fc.impl.util.io.MessageChannel.getIncomingMessageLength(MessageChannel.java:92)
            at com.documentum.fc.impl.util.io.MessageChannel.read(MessageChannel.java:77)
            - locked <0x000000061f77edb8> (a com.documentum.fc.impl.util.io.MessageChannel)
            at com.documentum.fc.client.impl.connection.netwise.AbstractNetwiseRpcClient.receiveMessage(AbstractNetwiseRpcClient.java:183)
            at com.documentum.fc.client.impl.connection.docbase.netwise.NetwiseDocbaseRpcClient.applyForObject(NetwiseDocbaseRpcClient.java:638)
            - locked <0x000000061f77edd8> (a com.documentum.fc.client.impl.connection.docbase.netwise.NetwiseDocbaseRpcClient)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection$8.evaluate(DocbaseConnection.java:1372)
            - locked <0x000000061f77ee28> (a com.documentum.fc.client.impl.connection.docbase.DocbaseConnection)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.evaluateRpc(DocbaseConnection.java:1131)
            - locked <0x000000061f77ee28> (a com.documentum.fc.client.impl.connection.docbase.DocbaseConnection)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.applyForObject(DocbaseConnection.java:1364)
            at com.documentum.fc.client.impl.docbase.DocbaseApi.authenticateUser(DocbaseApi.java:1867)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.authenticate(DocbaseConnection.java:436)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.authenticateConnection(DocbaseConnectionManager.java:356)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.assignConnection(DocbaseConnectionManager.java:202)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnectionManager.getDocbaseConnection(DocbaseConnectionManager.java:105)
            at com.documentum.fc.client.impl.session.SessionFactory.newSession(SessionFactory.java:23)
            at com.documentum.fc.client.impl.session.PrincipalAwareSessionFactory.newSession(PrincipalAwareSessionFactory.java:44)
            at com.documentum.fc.client.impl.session.PooledSessionFactory.newSession(PooledSessionFactory.java:49)
            at com.documentum.fc.client.impl.session.SessionManager.getSessionFromFactory(SessionManager.java:134)
            at com.documentum.fc.client.impl.session.SessionManager.newSession(SessionManager.java:72)
            at com.documentum.fc.client.impl.session.SessionManager.getSession(SessionManager.java:191)
            - locked <0x0000000619d5a840> (a com.documentum.fc.client.impl.session.SessionManager)
            at com.documentum.web.formext.docbase.FormatService.queryFormats(FormatService.java:529)
            at com.documentum.web.formext.docbase.FormatService.getFormats(FormatService.java:581)
            - locked <0x00000005edaa41d8> (a java.lang.String)
            at com.documentum.web.formext.docbase.FormatService.getFormatsByFileExtension(FormatService.java:293)
            at com.documentum.web.formext.docbase.FormatService.guessFormatFromFileExtension(FormatService.java:376)
            at com.documentum.web.formext.docbase.FormatService.guessFormatFromFileExtension(FormatService.java:357)
            at com.documentum.web.servlet.FileFormatIconResolver.service(FileFormatIconResolver.java:122)
    

    I have no idea why, but EMC employees love to use following anti-pattern:

    synchronized (shared resource) {
      do some docbase stuff w/o
      exclusive access to docbase session
    }
    

    for that reason we have replaced a plenty of Webtop classes by our own non-blockingcorrect implementation, and that was exactly this case – wrong pattern. BUT, there was also unusual thing: as it is obvious from stack problem thread got stuck on authentication (“DocbaseConnection.authenticate(DocbaseConnection.java:436)”), typically such operation does not take a lot of time, but weblogic had marked thread as stuck, which in turn means that thread hasn’t changed it’s state for at least last 15 minutes (actual period was 15 hours :)). Sounds weird, doesn’t it? So, what is the issue? I have spent about a day to create a semi-reproducible testcase:

    /**
     * @author Andrey B. Panfilov <andrey@panfilov.tel>
     */
    public class Test implements Runnable {
    
    	private static final Map<Integer, Thread> THREADS = new HashMap<Integer, Thread>();
    
    	private static final List<AtomicLong> COUNTERS = new ArrayList<AtomicLong>();
    
    	public static final IDfClient CLIENT;
    
    	public static final int THREAD_COUNT = 20;
    
    	public static final AtomicInteger INPROGRESS = new AtomicInteger(THREAD_COUNT);
    
    	public static final Semaphore SEMAPHORE = new Semaphore(THREAD_COUNT);
    
    	static {
    		try {
    			CLIENT = new DfClientX().getLocalClient();
    		} catch (DfException ex) {
    			throw new RuntimeException(ex);
    		}
    	}
    
    	private int _threadNo;
    
    	public Test(int threadNo) {
    		_threadNo = threadNo;
    	}
    
    	public static void main(String[] args) throws Exception {
    
    		SEMAPHORE.acquire(THREAD_COUNT);
    
    		for (int i = 0; i < THREAD_COUNT; i++) {
    			Thread t = new Thread(new Test(i));
    			THREADS.put(i, t);
    			COUNTERS.add(new AtomicLong(0));
    			t.start();
    		}
    
    		while (INPROGRESS.get() > 0) {
    			Thread.sleep(1000);
    		}
    
    		SEMAPHORE.release(THREAD_COUNT);
    
    		int it = 0;
    		while (true) {
    			it++;
    			long[] l = new long[COUNTERS.size()];
    			for (int i = 0, n = COUNTERS.size(); i < n; i++) {
    				l[i] = COUNTERS.get(i).get();
    			}
    			Thread.sleep(10000);
    			int alive = 0;
    			int stuck = 0;
    			for (int i = 0, n = COUNTERS.size(); i < n; i++) {
    				Thread t = THREADS.get(i);
    				boolean isAlive = t.isAlive();
    				if (isAlive) {
    					alive++;
    					long val = COUNTERS.get(i).get();
    					if (val == l[i]) {
    						stuck++;
    					}
    				}
    			}
    			System.out.println("Iteration: " + it + ", alive: " + alive + ", stuck: " + stuck);
    			if (alive == 0) {
    				return;
    			}
    		}
    
    	}
    
    	@Override
    	public void run() {
    		IDfSession session = null;
    		try {
    			IDfSessionManager sessionManager = CLIENT.newSessionManager();
    			DfLoginInfo loginInfo = new DfLoginInfo("dmadmin", "dmadmin");
    			loginInfo.setForceAuthentication(false);
    			loginInfo.setPeriodicAuthentication(false);
    			sessionManager.setIdentity("DCTM_DEV", loginInfo);
    			session = sessionManager.getSession("DCTM_DEV");
    			INPROGRESS.decrementAndGet();
    			SEMAPHORE.acquire();
    			while (true) {
    				doJob(session);
    				COUNTERS.get(_threadNo).incrementAndGet();
    			}
    		} catch (Exception ex) {
    			// ignore
    			// do not disconnect because we need to count time properly
    		}
    	}
    
    	public static void doJob(IDfSession session) throws DfException {
    		IDfQuery query = new DfQuery("SELECT user_name FROM dm_user ENABLE(RETURN_TOP 10)");
    		IDfCollection collection = query.execute(session, IDfQuery.DF_EXEC_QUERY);
    		while (collection.next()) {
    			collection.getString("user_name");
    		}
    		// no finally block because we need to count time properly
    		collection.close();
    	}
    
    }
    

    which requires a bit tricky execution:

    1. start testcase
    2. after 10-20 seconds perform a hard shutdown of CS host (echo o > /proc/sysrq-trigger)
    3. if after 20 minutes testcase reports that there are stuck threads:
      Iteration: 114, alive: 2, stuck: 2
      Iteration: 115, alive: 2, stuck: 2
      Iteration: 116, alive: 2, stuck: 2
      Iteration: 117, alive: 2, stuck: 2
      Iteration: 118, alive: 2, stuck: 2

      you caught it, congratulations!

    Now the interesting thing is when testcase got reproduced we have following execution stack:

    "Thread-26" prio=10 tid=0x00007f84682f9000 nid=0x15d06 runnable [0x00007f84617bd000]
       java.lang.Thread.State: RUNNABLE
            at sun.nio.ch.FileDispatcher.read0(Native Method)
            at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
            at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
            at sun.nio.ch.IOUtil.read(IOUtil.java:171)
            at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
            - locked <0x00000000f20e45e8> (a java.lang.Object)
            at com.documentum.fc.impl.util.io.MessageChannel.readSocket(MessageChannel.java:123)
            at com.documentum.fc.impl.util.io.MessageChannel.readLength(MessageChannel.java:100)
            at com.documentum.fc.impl.util.io.MessageChannel.getIncomingMessageLength(MessageChannel.java:92)
            at com.documentum.fc.impl.util.io.MessageChannel.read(MessageChannel.java:77)
            - locked <0x00000000f242b850> (a com.documentum.fc.impl.util.io.MessageChannel)
            at com.documentum.fc.client.impl.connection.netwise.AbstractNetwiseRpcClient.receiveMessage(AbstractNetwiseRpcClient.java:183)
            at com.documentum.fc.client.impl.connection.docbase.netwise.NetwiseDocbaseRpcClient.applyForCollection(NetwiseDocbaseRpcClient.java:414)
            - locked <0x00000000f242b7c8> (a com.documentum.fc.client.impl.connection.docbase.netwise.NetwiseDocbaseRpcClient)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection$3.evaluate(DocbaseConnection.java:1277)
            - locked <0x00000000f23f5e08> (a com.documentum.fc.client.impl.connection.docbase.DocbaseConnection)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.evaluateRpc(DocbaseConnection.java:1139)
            - locked <0x00000000f23f5e08> (a com.documentum.fc.client.impl.connection.docbase.DocbaseConnection)
            at com.documentum.fc.client.impl.connection.docbase.DocbaseConnection.applyForCollection(DocbaseConnection.java:1265)
            at com.documentum.fc.client.impl.docbase.DocbaseApi.exec(DocbaseApi.java:83)
            at com.documentum.fc.client.impl.session.Session.query(Session.java:3630)
            at com.documentum.fc.client.impl.session.SessionHandle.query(SessionHandle.java:2322)
            at com.documentum.fc.client.DfQuery.runQuery(DfQuery.java:167)
            at com.documentum.fc.client.DfQuery.execute(DfQuery.java:216)
            at Test.doJob(Test.java:123)
            at Test.run(Test.java:112)
            at java.lang.Thread.run(Thread.java:662)
    

    and following connection state from OS perspective:

    [dmadmin@docu72dev02 ~]$ netstat -na --timers| sed -e '1,2p;/10000/!d'
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       Timer
    tcp        0      0 ::ffff:172.16.49.138:47424  ::ffff:172.16.49.131:10000  ESTABLISHED off (0.00/0/0)
    tcp        0      0 ::ffff:172.16.49.138:47406  ::ffff:172.16.49.131:10000  ESTABLISHED off (0.00/0/0)
    

    and this is weird: operating system thinks that connections to CS are idle, but java waits for CS response!. Looks like a bug between java and operating system. Note that it is much simpler to reproduce testcase on SSL-connections – in case of plain connections I rarely get one or two stuck sockets, but in case of SSL even the first attempt created a plenty of them:

    [dmadmin@docu72dev02 ~]$  netstat -na --timers| sed -e '1,2p;/10001/!d'
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       Timer
    tcp        0      0 ::ffff:172.16.49.138:55960  ::ffff:172.16.49.131:10001  ESTABLISHED off (0.00/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55951  ::ffff:172.16.49.131:10001  ESTABLISHED off (0.00/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55954  ::ffff:172.16.49.131:10001  ESTABLISHED off (0.00/0/0)
    tcp        0    122 ::ffff:172.16.49.138:55958  ::ffff:172.16.49.131:10001  ESTABLISHED on (73.96/10/0)
    tcp        0    122 ::ffff:172.16.49.138:55961  ::ffff:172.16.49.131:10001  ESTABLISHED on (70.89/10/0)
    tcp        0      0 ::ffff:172.16.49.138:55957  ::ffff:172.16.49.131:10001  ESTABLISHED off (0.00/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55959  ::ffff:172.16.49.131:10001  ESTABLISHED off (0.00/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55950  ::ffff:172.16.49.131:10001  ESTABLISHED off (0.00/0/0)
    tcp        0    314 ::ffff:172.16.49.138:55949  ::ffff:172.16.49.131:10001  ESTABLISHED on (72.94/10/0)
    tcp        0      0 ::ffff:172.16.49.138:55967  ::ffff:172.16.49.131:10001  ESTABLISHED off (0.00/0/0)
    tcp        0    122 ::ffff:172.16.49.138:55955  ::ffff:172.16.49.131:10001  ESTABLISHED on (71.91/10/0)
    tcp        0    122 ::ffff:172.16.49.138:55952  ::ffff:172.16.49.131:10001  ESTABLISHED on (69.87/10/0)
    tcp        0    122 ::ffff:172.16.49.138:55965  ::ffff:172.16.49.131:10001  ESTABLISHED on (70.89/10/0)
    tcp        0    122 ::ffff:172.16.49.138:55948  ::ffff:172.16.49.131:10001  ESTABLISHED on (70.89/10/0)
    tcp        0    122 ::ffff:172.16.49.138:55964  ::ffff:172.16.49.131:10001  ESTABLISHED on (70.89/10/0)
    tcp        0      0 ::ffff:172.16.49.138:55956  ::ffff:172.16.49.131:10001  ESTABLISHED off (0.00/0/0)
    tcp        0    122 ::ffff:172.16.49.138:55962  ::ffff:172.16.49.131:10001  ESTABLISHED on (71.91/10/0)
    tcp        0    122 ::ffff:172.16.49.138:55953  ::ffff:172.16.49.131:10001  ESTABLISHED on (70.89/10/0)
    tcp        0    122 ::ffff:172.16.49.138:55963  ::ffff:172.16.49.131:10001  ESTABLISHED on (71.91/10/0)
    tcp        0      0 ::ffff:172.16.49.138:55966  ::ffff:172.16.49.131:10001  ESTABLISHED off (0.00/0/0)
    

    Now let’s read EMC’s documentation (I previously already criticized EMC’s whitepaper):

    DFC application hangs on the Linux machine

    If the client machine running the DFC client application is a Linux machine, and if the server to which it is connected goes off the network, the socket takes very long time to timeout, Occasionally, it causes the application server on the DFC client application to stop responding. This problem is related to low-level operating system socket handling. The developer must change the operating system level networking kernel settings and enable dfc.session.keepalive.enable property in the dfc.properties file to resolve this problem. It is also recommended that application administrators collaborate with server administrators before making the change. Server administrators must indetify appropriate settings for the kernel parameters based on their perfromance requirement and deployment.

    Operating system level settings for Linux:

    • net.ipv4.tcp_keepalive_time = 1
    • net.ipv4.tcp_keepalive_intvl = 5
    • net.ipv4.tcp_keepalive_probes = 3
    • net.ipv4.tcp_retries2 = 3

    What is the problem there? At first, the value of the last parameter (net.ipv4.tcp_retries2) EMC suggests is wrong – this parameter is responsible for RTO retransmissions and it is irrelevant to the situation where “the socket takes very long time to timeout” – 15 minutes is not a “long time”, below is a full description of this parameter:

    tcp_retries2 - INTEGER
        This value influences the timeout of an alive TCP connection,
        when RTO retransmissions remain unacknowledged.
        Given a value of N, a hypothetical TCP connection following
        exponential backoff with an initial RTO of TCP_RTO_MIN would
        retransmit N times before killing the connection at the (N+1)th RTO.
    
        The default value of 15 yields a hypothetical timeout of 924.6
        seconds and is a lower bound for the effective timeout.
        TCP will effectively time out at the first RTO which exceeds the
        hypothetical timeout.
    
        RFC 1122 recommends at least 100 seconds for the timeout,
        which corresponds to a value of at least 8.
    

    At second, manipulations with keepalives look like an attempt to not solve a problem but create a rude hack: java/operating system does not fire a timer for our connections, let introduce keepalive timeouts instead. However such setting does really work – in case of enabled dfc.session.keepalive.enable I get following connections state:

    [dmadmin@docu72dev02 ~]$  netstat -na --timers| sed -e '1,2p;/10001/!d'
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       Timer
    tcp        0      0 ::ffff:172.16.49.138:55988  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55991  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55999  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7143.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55996  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55994  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0    314 ::ffff:172.16.49.138:55989  ::ffff:172.16.49.131:10001  ESTABLISHED on (3.09/5/0)
    tcp        0      0 ::ffff:172.16.49.138:55998  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7143.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55986  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:56003  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7143.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55997  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:56000  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7143.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55992  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0    314 ::ffff:172.16.49.138:55985  ::ffff:172.16.49.131:10001  ESTABLISHED on (3.11/5/0)
    tcp        0      0 ::ffff:172.16.49.138:55990  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:56001  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7143.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:56002  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7143.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55984  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.36/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55993  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55995  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    tcp        0      0 ::ffff:172.16.49.138:55987  ::ffff:172.16.49.131:10001  ESTABLISHED keepalive (7142.37/0/0)
    

    which promises that operating system will down stuck connections after (tcp_keepalive_time + tcp_keepalive_probes * tcp_keepalive_intvl) seconds.

    Tricking installer

    Last docker post inspired to me to write this one.
    Well, imagine that I want to setup dev environment, but during installation I faced with dumb problems:

    • I want to install Oracle 11g, but the only publicly available version is 11.2.0.1 (or 11.2.0.2 if I want to install XE)
    • Documentum installer puts some restrictions on database version:

    The solution is:

    • after you get error from installer find minimumDbVersions.ini file in /tmp directory
    • modify it to reflect actual DB version

    UPD

    Got asked two similar questions, the most thorough was:

    got a VM running with CS 7.2 and Oracle XE (from my install notes):

    1. On the “enter license screen”
    2. drop to a shell
    3. fix /opt/documentum/product/7.2/bin/dmdbtest to contain only (w/o quotes): “exit 0”

    then proceed the install

    dmdbtest performs both version and connectivity checks, when you replace dmdbtest by your own implementation you either need to implement connectivity test or make sure that everything works well, “replace dmdbtest by “exit 0″ scenario” is not a “general-purpose recommendation”.

    Dynamic and non-dynamic DFC preferences

    Dynamic preferences are preferences which could be modified in runtime via modifying dfc.properties file, and these preferences are:

    groovy:000> s = com.documentum.fc.common.DfPreferences.instance
    groovy:000> for (int i=0;i<s.attrCount;i++) {name=s.getAttr(i).name;  if(s.getPreference(name).dynamic && !s.getPreference(name).deprecated) {println name}}
    dfc.application_code
    dfc.batch_hint_size
    dfc.search.batch_hint_size
    dfc.folder.hint.docbase
    dfc.folder.hint.index
    dfc.codepage
    dfc.date_format
    dfc.time_zone
    dfc.verify_registration
    dfc.bocs.avail.refresh_interval
    dfc.acs.check_availability
    dfc.acs.config.refresh_interval
    dfc.acs.gr.refresh_interval
    dfc.bocs.check.keep_number
    dfc.cache.enable_persistence
    dfc.cache.format.currency_check_interval
    dfc.cache.store.currency_check_interval
    dfc.cache.type.currency_check_interval
    dfc.compatibility.truncate_long_values
    dfc.compatibility.return_null_when_no_values
    dfc.compatibility.preserve_session_info_messages
    dfc.data.local_dir
    dfc.content.castore.max_write_attempts
    dfc.content.castore.write_sleep_interval
    dfc.content.extern_store_content_static
    dfc.diagnostics.exception.throw_warnings
    dfc.diagnostics.exception.include_stack
    dfc.diagnostics.preference.warn_about_rename
    dfc.diagnostics.preference.warn_about_deprecation
    dfc.diagnostics.warn_about_dmcl_call
    dfc.docbroker.host
    dfc.docbroker.port
    dfc.docbroker.protocol
    dfc.docbroker.service
    dfc.docbroker.timeout
    dfc.docbroker.exclude.failure_threshold
    dfc.docbroker.exclusion.time
    dfc.exception.include_decoration
    dfc.exception.include_id
    dfc.globalregistry.username
    dfc.globalregistry.password
    dfc.globalregistry.repository
    dfc.globalregistry.domain
    dfc.logging.level_to_force_stack
    dfc.logging.verbose
    dfc.machine.id
    dfc.relationship.schema_expiration_interval
    dfc.valueassistance.refresh.interval
    dfc.search.external_sources.enable
    dfc.search.external_sources.host
    dfc.search.external_sources.password
    dfc.search.external_sources.port
    dfc.search.external_sources.ssl.enable
    dfc.search.external_sources.ssl.keystore
    dfc.search.external_sources.ssl.keystore_password
    dfc.search.external_sources.request_timeout
    dfc.search.external_sources.username
    dfc.search.external_sources.backup.port
    dfc.search.external_sources.backup.host
    dfc.search.external_sources.retry.period
    dfc.search.fulltext.enable
    dfc.search.max_results
    dfc.search.max_results_per_source
    dfc.search.formatcache.refresh_interval
    dfc.search.sourcecache.refresh_interval
    dfc.search.typecache.refresh_interval
    dfc.search.eos.mappingcache.refresh_interval
    dfc.search.xquery.generation.enable
    dfc.search.xquery.option.parallel_execution.enable
    dfc.search.xquery.option.parallel_summary_computing.enable
    dfc.search.matching_terms_computing.enable
    dfc.dqlhints.file
    dfc.searchranking.file
    dfc.security.keystore.password
    dfc.security.keystore.privatekey.password
    dfc.session.dynamic_delay
    dfc.connection.queuing_threshold
    dfc.session.max_deadlock_retries
    dfc.session.max_error_retries
    dfc.session.max_connect_retries
    dfc.session.pool.enable
    dfc.session.pool.expiration_interval
    dfc.connection.reuse_time_limit
    dfc.session.reuse_limit
    dfc.session.pool.mode
    dfc.session.secure_connect_default
    dfc.session.allow_trusted_login
    dfc.session.pool.reauthenticate_interval
    dfc.compatibility.allow_weak_disconnect
    dfc.compatibility.useD7SessionPooling
    dfc.session.surrogate.check_interval
    dfc.session.surrogate.mode
    dfc.session.load_balance_strategy
    dfc.session.max_server_choice_age
    dfc.storagepolicy.enable
    dfc.tokenstorage.dir
    dfc.tokenstorage.enable
    dfc.session.keepalive.enable
    dfc.tracing.enable
    dfc.tracing.mode
    dfc.tracing.verbose
    dfc.tracing.file_creation_mode
    dfc.tracing.timing_style
    dfc.tracing.max_stack_depth
    dfc.tracing.thread_name_filter
    dfc.tracing.user_name_filter
    dfc.tracing.method_name_filter
    dfc.tracing.print_stack_on_method_match
    dfc.tracing.include_rpcs
    dfc.tracing.include_session_id
    dfc.tracing.include_rpc_count
    dfc.tracing.print_exception_stack
    dfc.tracing.date_column_width
    dfc.tracing.date_format
    dfc.tracing.dir
    dfc.tracing.log.category
    dfc.tracing.log.level
    dfc.tracing.log.additivity
    dfc.tracing.file_prefix
    dfc.tracing.file_override
    dfc.tracing.max_backup_index
    dfc.tracing.max_file_size
    dfc.xml.encode_special_chars_in_attrs
    dfc.xml.suppress_default_namespace_decl
    dfc.ldap.bof.validate_password
    dfc.lwo.allow_propagating_changes
    dfc.lwo.log_materialize_stack
    dfc.batchmanager.max_batch_size
    dfc.storage.filestore_accelerator.enable
    dfc.crypto.repository
    dfc.connection.unused_connection_timeout
    dfc.connection.reuse_threshold
    dfc.connection.connection_reuse_matching_user_bias
    dfc.connection.cleanup_check_window
    dfc.connection.statistics_history_count
    dfc.connection.profile_connections
    dfc.connection.trace_connections_only
    dfc.connection.queue_wait_time
    dfc.connection.out_of_sessions_wait_time
    dfc.license.cache_expiry_interval
    dfc.license.server_reconnect_interval
    dfc.license.metrics_log_frequency
    

    in order to change non-dynamic preference you need to restart JVM, these preferences are:

    groovy:000> s = com.documentum.fc.common.DfPreferences.instance
    groovy:000> for (int i=0;i<s.attrCount;i++) {name=s.getAttr(i).name;  if(!s.getPreference(name).dynamic && !s.getPreference(name).deprecated) {println name}}
    dfc.name
    dfc.config.dir
    dfc.config.file
    dfc.data.dir
    dfc.data.user_dir
    dfc.data.checkout_dir
    dfc.data.export_dir
    dfc.data.local_diskfull_limit
    dfc.data.local_diskfull_check_interval
    dfc.data.local_clean_on_init
    dfc.data.local_purge_on_diskfull
    dfc.registry.mode
    dfc.permit.grant.write.user_default_folder
    dfc.registry.file
    dfc.config.check_interval
    dfc.query.object_name_for_docbase
    dfc.query.should_include_object_name
    dfc.locale
    dfc.version
    dfc.acs.avail.refresh_interval
    dfc.acs.protocol.handler.class_name
    dfc.acs.protocol.handler.protocol
    dfc.acs.request.expiration_interval
    dfc.admin.ldif_file_charset
    dfc.appledouble.resource_file_ext
    dfc.bocs.check.http_method
    dfc.bof.cache.append_name
    dfc.bof.cache.currency_check_interval
    dfc.bof.cache.cleanup_interval
    dfc.bof.cache.enable_preload
    dfc.bof.classloader.enable_extension_loader_first
    dfc.cache.dir
    dfc.cache.write_interval
    dfc.cache.ddinfo.size
    dfc.cache.object.size
    dfc.cache.query.size
    dfc.cache.type.size
    dfc.cache.currency_scoping.enable
    dfc.cache.currency_scoping.query_result_max_count
    dfc.data.umask
    dfc.content.use_compression
    dfc.content.use_content_server
    dfc.diagnostics.resources.enable
    dfc.docbroker.auto_request_forward
    dfc.docbroker.search_order
    dfc.docbroker.debug.docbase_id
    dfc.docbroker.debug.host
    dfc.docbroker.debug.port
    dfc.docbroker.debug.service
    dfc.kerberos.mapped_docbaseSPN
    dfc.kerberos.docbaseName
    dfc.globalregistry.connect_attempt_interval
    dfc.object.id_batch_size
    dfc.privilege.enable
    dfc.privilege.use_anonymous_hostname
    dfc.reference.binding_label
    dfc.validation.allow.empty_string.list_complete
    dfc.search.docbase.broker_count
    dfc.search.external_sources.adapter.domain
    dfc.search.external_sources.broker_count
    dfc.search.external_sources.rmi_name
    dfc.security.keystore.file
    dfc.session.max_collection_count
    dfc.session.max_count
    dfc.session.connect_timeout
    dfc.session.connect_retry_interval
    dfc.session.global_pool_enabled
    dfc.aspect.virtual_default_aspect_supported
    dfc.storagepolicy.ignore_rule_errors
    dfc.storagepolicy.validation_interval
    dfc.validation.expr.currency_check_interval
    dfc.validation.expr.debug.all
    dfc.validation.expr.debug.eval
    dfc.validation.expr.debug.code
    dfc.validation.expr.debug.tree
    dfc.validation.expr.disable_java
    dfc.validation.expr.load_native_library
    dfc.validation.overrides.currency_check_interval
    dfc.validation.overrides.enable
    dfc.vdm.max_child_flush_count
    dfc.xml.retain_early_binding_to_current
    dfc.xml.record_inline_descendants
    dfc.xml.use_strict_uri
    dfc.jmx.connector.port
    dfc.jmx.connector.readonly.user
    dfc.jmx.connector.readonly.password
    dfc.jmx.connector.readwrite.user
    dfc.jmx.connector.readwrite.password
    dfc.jmx.mbeanserver.domain
    dfc.jmx.mbean_server_provider
    dfc.operations.customization.version
    dfc.content.has_store_access
    dfc.housekeeping.interval
    dfc.housekeeping.viewed_file_retention_period
    dfc.internal.purge_far_connections
    dfc.license.core_thread_count
    dfc.license.max_queue_size
    dfc.license.max_cache_size
    dfc.security.ssl.truststore
    dfc.security.ssl.truststore_password
    dfc.security.ssl.use_existing_truststore
    

    Docker and Documentum. Part III

    If my memory serves me right, last time I promised to demonstrate how it is possible to maintain networking in Docker environment, but didn’t keep that promise. Actually, I realised that using any kind of external tools to maintain networking in Docker environment is completely impractical and decided to wait for native support of “static” ip-address assignments in Docker. And finally, Docker team released this capability in 1.10. Cool.

    If you are going to use Docker it is good idea to consider following suggestions:

    Surprisingly, Documentum installer 7.2 does not support Oracle XE (latest Oracle XE version is 11.2.0.2, lowest Oracle version supported by Documentum installer 7.2 is 11.2.0.3), hopefully, I was able to manage this mistake.

    Enjoy.

    Encryption madness. Part III

    On last week my colleagues were complaining about slow JMS performance on specific case: as a part of workflow they process tiff and pdf documents on JMS to place graphic stamp into content, and on 6.7SP1 everything was working smoothly, but on 7.2 this operation started taking a lot of time. The problem was following: during image processing they (i.e. my colleagues) create temporary files using File.createTempFile() method, ImageIO API in java also utilises File.createTempFile() method, so, I had counted about dozen of File.createTempFile() calls during one stamping operation, File.createTempFile() method is based on java.security.SecureRandom which is extremely slow in D7.2 environment, for example, following code

    public class Test {
    
    	public static final AtomicLong COUNTER = new AtomicLong(0);
    
    	public static void main(String[] args) throws Exception {
    
    		for (int i = 0; i < 100; i++) {
    			new Thread(new Runnable() {
    				@Override
    				public void run() {
    					try {
    						while (true) {
    							File.createTempFile("xxx", "xxx").delete();
    							COUNTER.incrementAndGet();
    						}
    					} catch (IOException ex) {
    						throw new RuntimeException(ex);
    					}
    				}
    			}).start();
    		}
    
    		while (true) {
    			long l = COUNTER.get();
    			Thread.sleep(10000);
    			System.out.println(COUNTER.get() - l);
    		}
    
    	}
    
    }
    

    demonstrates 100 times slower performance on JVM shipped with Content Server:

    [dmadmin@docu72dev01 ~]$ type java
    java is hashed (/u01/documentum/cs/shared/java64/1.7.0_72/bin/java)
    [dmadmin@docu72dev01 ~]$ java Test
    4302
    5173
    [dmadmin@docu72dev01 ~]$ export JAVA_HOME=/opt/java/jdk/sun/x86_64/jdk1.7.0_75
    [dmadmin@docu72dev01 ~]$ $JAVA_HOME/bin/java Test
    503042
    518592
    

    What the hell is going on? The answer is simple: EMC embedded extremely slowFIPS 140-2 enabled random number generator into JVM