Ban BOF2 deployment

Two weeks ago I tried to run Webtop 6.7SP1 against Content Server 7.2 (yeap, I do know that such configuration is considered as “unsupported”, but my opinion is EMC is just not familiar with product they are trying to developkill) and got an exception when I was trying to launch workflow:

java.lang.IllegalAccessError: tried to access method 
     from class com.documentum.fc.client.DfWorkflow
at com.documentum.fc.client.DfWorkflow.addPackageInternal(
at com.documentum.fc.client.DfWorkflow.addPackage(
at com.documentum.bpm.impl.DfWorkflowEx___PROXY.addPackage(
at com.documentum.fc.client.DfWorkflowBuilder.addPackage(

For every java developer this error looks weird because DfWorkitem#checkComponentTypes method has a package access level and both DfWorkflow and DfWorkitem classes reside in the same package. However, I do know that when DFC loads BOF2 modules (both dm_workflow and dmi_workitem are now deployed as BOF2) it performs some manipulations with bytecode and classloader and these manipulations break a lot of things, so the rule of thumb when developing TBO is never ever use static methods inside TBO classes – if you need static method put it into utility class, unfortunately, EMC coders do not know about this rule, so we are doomed to investigate stupid exceptions.

How to fix my error? After a little deliberation I decided that the best option is to somehow hide these modules (dm_worklow and dmi_workitem) from DFC 6.7SP1. Initially I thought that min_dfc_version attribute in dmc_module is exactly what I was need:

though description of this attribute is very obscure:

my understanding was: “if certain module is not designed for specific DFC version, DFC must not load it at all” – it is quite logical, isn’t it? Nothing like this! EMC coders are adepts of special kind of logic which is incomprehensible for other humans – changing value of min_dfc_version does not affect DFC behavior. What is next? Is it possible to create a multiple modules and allow DFC to “pick up” the correct one? Actually, I was need that DFC do not pick up any, because in that case it should switch to or IntrinsicModuleRegistry – seems not to be logical, but I did comprehend incomprehensible logic of EMC developers 🙂 Let’s try:

API> create,c,dm_folder
API> set,c,l,object_name
SET> 1
API> link,c,l,/System/Modules/TBO
API> save,c,l
API> create,c,dmc_module
API> set,c,l,object_name
SET> dm_workflow
API> set,c,l,a_module_type
API> link,c,l,/System/Modules/TBO/1
API> save,c,l
[DFC_BOF_MUST_HAVE_UNIQUE_NAMES] Each business object must have a unique name. 
   Name "dm_workflow" already exists.

Damn! Where is my RPC trace?

API> retrieve,c,dm_workflow


Object Id: 0000000000000000
Argument Object:
OBJ NULL 0 0 0 0
A 169 select r_object_id from dmc_module where a_module_type='TBO'  
   and (object_name='dm_workflow' or object_name like 'dm_workflow..%' ) 
   and folder('/System/Modules',descend)

Aha, DFC tries to find not only modules with the same name as type name but also modules with name starting with ‘type_name..’, interesting … Complete solution:

API> retrieve,c,dmc_module where object_name='dm_workflow'
API> set,c,l,min_dfc_version
SET> 7.1.0000.0000
API> save,c,l
API> create,c,dmc_module
API> set,c,l,object_name
SET> dm_workflow..neverload
--- I believe that DFC will never
--- get 9.9 version
API> set,c,l,min_dfc_version
SET> 9.9.0000.0000
API> set,c,l,a_module_type
API> link,c,l,/System/Modules/TBO
API> save,c,l


Q & A. IX


i’m relatively new to the xCP2 (and documentum) world and i have a question: the color of the application’s background is blue and i don’t like blue… juste joking !

i couldn’t find anywhere tips about continuous integration. is there a way to build and xCP2 project with commande line ? i want to make a continuous integration using Jenkins, SOMETHING to build, and (i think)the xmstool to deploy it every night.
However, couldn’t find anything to build the project from command line(and i shearshed up to google page 5!) in a unix environnement. Do you have any clues about what i can use?



I’m not sure that I’ll able to answer your question because I have never tried to perform automatic builds of xCP2 projects before (actually, after some unsuccessful attempts to create a complex (not a “hello world”) xCP2 application we have found that at current moment xCP2 is completely unusable/unstable/unreliable/undocumented and switched back to WDK/DFC), however EMC folks tell that there is an option to mavenize xCP2 project – it seems that attempt to execute “mvm package” is doomed to failure if xCP Designer is up and running:

> G:\app\maven\3.2.3\bin\mvn package -Dmaven.repo.local=G:\app\xCPDesigner\maven


[ERROR] Unexpected system error packaging dar for project 'ap'
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.120 s
[INFO] Finished at: 2015-07-30T21:34:08+10:00
[INFO] Final Memory: 18M/223M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.emc.xcp.builder:xcp-dar:1.0.8:run (xcp-dar) on project ap: 
   Xcp mojo executing command 'preparepackage' for project 'ap' failed unexpectedly: j
ava.lang.IllegalArgumentException: xcpProject must not be null.
[ERROR] at com.emc.xcp.builder.packaging.PackagerUtil.generateRunConfig(
[ERROR] at
[ERROR] at
[ERROR] at
[ERROR] at
[ERROR] at
[ERROR] at javax.servlet.http.HttpServlet.service(
[ERROR] at javax.servlet.http.HttpServlet.service(
[ERROR] at org.eclipse.equinox.http.registry.internal.ServletManager$ServletWrapper.service(
[ERROR] at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(
[ERROR] at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(
[ERROR] at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(
[ERROR] at javax.servlet.http.HttpServlet.service(
[ERROR] at org.eclipse.equinox.http.jetty.internal.HttpServerManager$InternalHttpServiceServlet.service(
[ERROR] at org.eclipse.jetty.servlet.ServletHolder.handle(
[ERROR] at org.eclipse.jetty.servlet.ServletHandler.doHandle(

but in offline mode it does build something (note designerPath property):

G:\app\maven\3.2.3\bin\mvn package -Dmaven.repo.local=G:\app\xCPDesigner\maven -DdesignerPath=G:\app\xCPDesigner


[INFO] Webapp assembled in [5613 msecs]
[INFO] Building war: G:\app\xCPDesigner\Applications\ap\ap\target\ap-ap-1.0.0.war
[WARNING] Warning: selected war files include a WEB-INF/web.xml which will be ignored
(webxml attribute is missing from war task, or ignoreWebxml attribute is specified as 'true')
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:02 min
[INFO] Finished at: 2015-07-30T21:36:41+10:00
[INFO] Final Memory: 15M/218M
[INFO] ------------------------------------------------------------------------

Fighting with DEV ENV

About a month ago I thought about upgrading my current T420s laptop to something more modern and powerful, unfortunately it have turned out that laptop market is unable to offer anything suitable for me, though I do not expect anything extraordinary, just:

So, I gave up the idea to buy new laptop and decided to save moneyspend money on booze, but such approach does not actually solve my problem: I need to run a couple of virtual machines with Documentum on my current laptop. What to do? The answer is obvious: take a problem and solve it. Below you can find some suggestions about how to decrease memory footprint for Documentum and increase response time – finally, I got something like:

Decrease number of concurrent jobs

By default agentexec executes up to three jobs in a polling cycle, some of docbase jobs are too heavy for being run simultaneously.

API> retrieve,c,dm_method where object_name='agent_exec_method'
API> get,c,l,method_verb
./dm_agent_exec -max_concurrent_jobs 1

Disable saving job’s logs into repository

Actually, this suggestion is very controversial and it may not work in some cases. The problem is vendor assumes that putting “ECM” label on software product automatically turns this product into scrap-heap, agentexec is a good confirmation of this point: every time when agentexec executes job it saves job’s log into repository, and, as per my memory, I had checked those logs may be four or five times during last eight years, why do not disable this useless feature? I believe that it’s fucking simple to add extra attribute to dm_job object to control behavior of agentexec, instead of that vendor created dumb job intended for clearing obsolete logs. Fortunately, when playing with dm_job’s method_trace_level attribute I have found that setting method_trace_level to -1 prevents agentexec from saving job’s log into repository, unfortunately, standard jobs does not recognize this value:

[com.documentum.mthdservlet.DoMethod] - Exception invoking com.documentum.bpm.method.XCPAutoTasKMgmt.
DfMethodArgumentException:: THREAD: http--; 
       MSG: [DFC_METHOD_BAD_ARGUMENT_VALUE] Argument method_trace_level 
       has an invalid value (4294967295); ERRORCODE: ff; NEXT: null
    at com.documentum.fc.methodserver.DfMethodArgumentException.invalidArgument(
    at com.documentum.fc.methodserver.DfMethodArgumentManager.getInt(
    at com.documentum.fc.methodserver.DfStandardJobArguments.<init>(
    at com.documentum.fc.methodserver.DfMethodArgumentManager.getJobArguments(
    at com.documentum.fc.methodserver.DfMethodArgumentManager.<init>(
    at com.documentum.bpm.Utils.GenericJobMethod.execute(
    at com.documentum.mthdservlet.DfMethodRunner.runIt(Unknown Source)
    at com.documentum.mthdservlet.AMethodRunner.runAndReturnStatus(Unknown Source)
    at com.documentum.mthdservlet.DoMethod.invokeMethod(Unknown Source)
    at com.documentum.mthdservlet.DoMethod.doPost(Unknown Source)
    at javax.servlet.http.HttpServlet.service(
    at javax.servlet.http.HttpServlet.service(

What to do? HexEditor to the rescue: I have replaced “-docbase_name %s -user_name %s -job_id %s -method_trace_level %s” pattern in dm_agent_exec binary by “-docbase_name %s -user_name %s -job_id %s -method_trace_level 0” and now agentexec does not save job’s logs into repository and jobs are happy with parsing 0 as trace level:

API> ?,c,select count(*) from dm_document where FOLDER('/Temp/Jobs',DESCEND)
(1 row affected)

Another problem with agentexec it loves to store log files with stupid names in $DOCUMENTUM/dba/log/<docbaseid>/agentexec directory:

agentexec]$ ls | grep save.

to solve this problem I replaced “%s.%s.%02d.%02d.%02d.%02d.%02d.%02d” pattern in dm_agent_exec binary by “%s.%s\0%02d.%02d.%02d.%02d.%02d.%02d” (\0 is a binary zero).

Disable useless jobs

By default, Documentum installer enables following jobs:

  • dm_usageReport
  • dm_WfmsTimer
  • dm_ContentWarning
  • dm_DBWarning
  • dm_StateOfDocbase
  • dm_UpdateStats
  • dm_DataDictionaryPublisher
  • dm_bpm_XCPAutoTaskMgmt
  • dce_Clean
  • dm_QmPriorityAging
  • dm_QmPriorityNotification
  • dm_QmThresholdNotification
  • dm_WFReporting
  • dm_WFSuspendTimer

and all of them are useless for development environment:

API> ?,c,update dm_job objects set is_inactive=TRUE 
   where object_name like 'dm\_%' escape '\' or object_name like 'dce\_%' escape '\'
(1 row affected)
[DM_QUERY_I_NUM_UPDATE]info:  "50 objects were affected by your UPDATE statement."

Improve startup time of DMCL applications


# java_options      - Options for the Java VM.
java_options = "-Xms4m -Xmx64m -XX:PermSize=4m -XX:MaxPermSize=256m -XX:+UseSerialGC -Xrs"

# List of providers and their preference orders (see above):

Improve startup time of Java Method Server

  1. undeploy acs.ear
  2. tune heap size – default settings: -Xms1024m -Xmx1024m -XX:PermSize=64m -XX:MaxPermSize=256m are too greedy
  3. replace standalone.xml with:
    <?xml version='1.0' encoding='UTF-8'?>
    <server xmlns="urn:jboss:domain:1.2">
            <extension module=""/>
            <extension module=""/>
            <extension module=""/>
            <extension module=""/>
            <extension module=""/>
            <extension module=""/>
            <subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
                <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="0" deployment-timeout="300"/>
            <subsystem xmlns="urn:jboss:domain:ee:1.0"/>
            <subsystem xmlns="urn:jboss:domain:naming:1.1"/>
            <subsystem xmlns="urn:jboss:domain:remoting:1.1"/>
            <subsystem xmlns="urn:jboss:domain:security:1.1">
                    <security-domain name="other" cache-type="default"/>
            <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
                <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
                <virtual-server name="default-host" enable-welcome-root="true">
                    <alias name="localhost"/>
                    <alias name=""/>
            <interface name="public">
                <inet-address value="${jboss.bind.address:}"/>
        <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
            <socket-binding name="ajp" port="9089"/>
            <socket-binding name="http" port="9080"/>
            <socket-binding name="https" port="9082"/>

Disable SSL

It seems that starting from 7.2(?) Documentum installer enables SSL by default:

API> retrieve,c,dm_server_config
API> get,c,l,secure_connect_mode

disable it to improve response time:

API> set,c,l,secure_connect_mode
SET> native
API> save,c,l

Disable email notifications


mail_notification = F

Disable updating last_login_utc_time attribute of dm_user objects

API> ?,c,exec exec_sql with 
    query='update dm_user_s set i_vstamp=i_vstamp+1, 
(1 row affected)

Disable updating r_access_date attribute of dm_sysobject objects


update_access_date = F

Disable MACL security

API> retrieve,c,dm_docbase_config
API> set,c,l,macl_security_disabled
API> save,c,l

Disable dmbasic method server


# This controls the dmbasic method server.
method_server_enabled = F
method_server_threads = 5

Disable auditing

API> unaudit,c,,dm_default_set
API> unaudit,c,,dm_logon_failure


Do you remember my post about backdoor in default installation and doubts about SDLC in EMC? This sad history has a sequel! About one week ago my skypemate asked me about whether I experienced difficulties with installing latest patchsets on Content Server, today I found post on ECN with similar problem:

It turns out, that EMC tried to fix security issue described in my post about backdoor in default installation and, as expected, got failed – latest Content Server patches contain special XSLT intended to remove mapping for DmSampleServlet from web.xml:

<xsl:stylesheet version="1.0" exclude-result-prefixes="xs"
    <xsl:template match="@* | node()">
            <xsl:apply-templates select="@* | node()"/>

    <xsl:template match="//servlet[servlet-name = 'DmSampleServlet']"/>
    <xsl:template match="//servlet-mapping[servlet-name = 'DmSampleServlet']"/>

I seems that EMC thinks customers who are not going to upgrade their pre-D7 installations do not deserve security fix.


Another piece of nonsense from EMC

Sometimes I do think that EMC specially hires people who unable to read and understand what they read. If you take a look at any EMC’s vacancy you will find something like:

Actually, I do want to believe that “Equal Employment Opportunity employer” relates only to sex, raсe and religion and irrelevant to mental disorders, but facts sow doubt in me.

Today EMC have published two funny articles.

The JSP specification requires that an attribute is preceded by whitespace:

Strange, EMC developers does not follow JSP specification, but Apache Tomcat is to blame because it does allow to relax specification checks. Moreover, I can’t understand how disabling HttpOnly cookies affects JSP specification – it seems that I have mental disorder :). Actually, HttpOnly cookies is a cool security feature which mitigates a lot of XSS attacks, so, why do not change behavior of UCF applet and make customers safer?

Another article:

is more strange because it suggests the opposite to what the administrator really needs to do:

Java and imaging

About one month ago I was involved in project where I was need to split/convert multipage TIFFs into PNGs. Actually, there is a weird situation: at first glance EMC Documentum supports storing content in multiple pages, on the other hand there are no client application which supports this capability, so in case of multipage images customers/developers prefer to use either TIFF or PDF format, for example, below is a format distribution from one of our repositories:

Format No of documents
pdf 11200148
tif 7746475
jpeg 1036117
msw12 555678
rtf 385219
phtml 260188
msw6 248655
zip 178097
excel8book 171200
excel12book 165109
msw8 140428
excel4sheet 114212
mdi 46387
jpeg_lres 34053
crtext 34008
rar 31569
bmp 18159
png 14029
7z 9470
xml 4719
excel 3472
excel12mebook 2750
rlx 2545
mht 2126
msw 1679
gif 1457


So, TIFF format is very popular, however even in 2015 support of TIFF format in Java is very limited – there is nothing to add to Harald Kuhr’s presentation:

Fortunately, I had found a cool java library, which has passable support of TIFF format.

How to slow down ingestion

On Friday when I was trying to insert about 200 thousand objects into repository I have noticed that ingestion rate is much slower than I was expecting, after creating a couple of thread dumps I have found out that during instantiation of new object DFC performs extra fetches from repository:

These fetches are caused by definition of attribute default values in object type – every definition of default value causes extra fetch from repository:

Dumb dmbasic

Today I had faced with following “challenge”:

When dealing with composer I prefer to put all object types into one project and all TBOs into another, due to following reasons:

  • both content server and composer are extremely slow in installing object types – I have seen some situations when installing object types takes about two hours because content server performs millions database selects
  • TBO implementation changes more often than object type definition

but today I was need to attach default aspects to certain object types, in order to do that I had written a post-install procedure (and, yeap, my eyes had came out):

Const glabel As String          = "Label"
Const ginfo As String           = "Info"
Const gerror As String          = "Error"

Private Sub PrintMessage(mssg As String, mssgtype As String)
  If(mssgtype=glabel) Then
            Print "<BR><B><FONT size=3>"
            Print mssg
            print "</FONT></B>"
  ElseIf(mssgtype=ginfo) Then
            Print "<BR><FONT color=blue>"
            Print mssg
            print "</FONT>"
  ElseIf(mssgtype=gerror) Then
            Print "<BR><FONT color=red size=3>"
            Print mssg
            print "</FONT>"
            Print "<BR>" & mssg
  End If
End Sub

Private Sub AddAspect(TypeName2 As String, AspectName As String)
    Dim AddQryAPI As String
    Dim APIStatus As String
    Dim Status As Integer
    Dim QueryID As String
    Dim AttributeID As String

    AddQryAPI = "query,c,ALTER TYPE " & TypeName2 & " ADD DEFAULT ASPECTS " & AspectName
    Call PrintMessage("Adding the aspect " & AspectName & " to " & TypeName2, ginfo)
    QueryID = dmAPIGet(AddQryAPI)
    If QueryID <> "" Then
         Call PrintMessage("Adding success", ginfo)
         Status = dmAPIExec("close,c," & QueryID)
         msgStr$ = dmAPIGet("getmessage,c")
         Call PrintMessage(msgStr$, gerror)
         msgStr$ = "Failed to add default aspect " & AspectName
         Call PrintMessage(msgStr$, gerror)
    End If
End Sub

Sub PostInstall(DocbaseName As String, UserName As String, Password As String)
  Dim SessionID As String

  SessionID= dmAPIGet("connect," & DocbaseName & "," & UserName & "," & Password)
  If SessionID ="" Then
    Print "Fail to connect to docbase " & DocbaseName &" as user " & UserName
    Print "Connect to docbase " & DocbaseName &" as user " & UserName
  End If

  Call AddAspect("type1", "aspect1")

End Sub

but when installing composer project I got weird error:

[FATAL]  An unexpected error has occurred during installation.

For more details, check the most recent error log located by default in 'C:\Temp\documentum\'.
com.emc.ide.installer.PostInstallException: Error running post-install procedure "postinstall"
        at com.emc.ide.installer.popup.actions.InstallOperation.installDar(InstallOperation.ja
        at org.eclipse.jface.operation.ModalContext$
Caused by: com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerException: Failed to run dm
mbasic\dmbasic.exe -f G:\Users\andrey\work\vk\ord_int\dev\ContentServer\ComposerProject\esed_d
        at com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerUtils.executeCommand(Proced
        at com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerUtils.executeDmBasic(Proced
        at com.emc.ide.external.dfc.procedurerunner.ProcedureRunner.execute(ProcedureRunner.ja
        ... 6 more
Caused by: Cannot run program "G:\app\emc\composer\7.2\plugins\
rror=740, The requested operation requires elevation
        at java.lang.ProcessBuilder.start(
        at com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerUtils.executeCommand(Proced
        ... 9 more
Caused by: CreateProcess error=740, The requested operation requires elev
        at java.lang.ProcessImpl.create(Native Method)
        at java.lang.ProcessImpl.<init>(
        at java.lang.ProcessImpl.start(
        at java.lang.ProcessBuilder.start(
        ... 10 more

What the fuck did just happened here?

For some weird reason EMC coders decided that dmbasic executable requires administrative privileges (note shield icon – I bet dmbasic sends some information directly to EMC, otherwise I can’t find any reason why it requires administrative privileges):

How to fix that? Initially I was trying to find manifest editor in google, but after a couple of futile attempts I just edited dmbasic binary in HEX editor and got following (note extra spaces intended to preserve file size):