Resurrection of Repoint

My colleagues used to like perform some routine Documentum operations through Repoint (not a best choice though), unfortunately now Repoint is not under “active development” and it does not work with modern JDK versions, hence it does not work properly with TBO compiled by modern JDK versions. Previously Alvaro de Andres shared an instructions how to plug Repoint into Composer (another one not best choice:)), but yesterday I discovered another project maintained by Karol Bryd: Documentum Repoint “resurrected” (check also: Repoint 0.2.0 released, Documentum Repoint Resurrected!), indeed Repoint has Resurrected:

Fighting with Composer. NLS data

Case: I played with locale data in composer, installed project a couple of times and now I want to perform “clean” installation without reinstalling the whole repository. The problem is: in composer locale data is hierarchical but in repository it is flat, for example, if I overwrote attribute’s label for subtype and now I want to inherit it from supertype again, I need to keep overwritten label in sync with supertype – seems not to be convenient.

Solution:

-- deleting NLS data for particular types
DELETE dmi_object_type
 WHERE r_object_id IN (SELECT r_object_id
                         FROM dm_nls_dd_info_s
                        WHERE parent_id IN (SELECT r_object_id
                                              FROM dm_aggr_domain_s
                                             WHERE type_name ...));

DELETE dm_nls_dd_info_r
 WHERE r_object_id IN (SELECT r_object_id
                         FROM dm_nls_dd_info_s
                        WHERE parent_id IN (SELECT r_object_id
                                              FROM dm_aggr_domain_s
                                             WHERE type_name ...));

DELETE dm_nls_dd_info_s
 WHERE parent_id IN (SELECT r_object_id
                       FROM dm_aggr_domain_s
                      WHERE type_name ...);

UPDATE dm_domain_r
   SET nls_dd_info = NULL, nls_key = NULL
 WHERE     r_object_id IN (SELECT r_object_id
                             FROM dm_aggr_domain_s
                            WHERE type_name ...)
       AND nls_dd_info IS NOT NULL;

UPDATE dm_domain_r
   SET nls_dd_info = NULL, nls_key = NULL
 WHERE     r_object_id IN (SELECT r_object_id
                             FROM dm_domain_s
                            WHERE parent_id IN (SELECT r_object_id
                                                  FROM dm_aggr_domain_s
                                                 WHERE type_name ...))
       AND nls_dd_info IS NOT NULL;

-- generating API commands to remove dd objects
SELECT 'apply,c,,REMOVE_DD_OBJECT,ID,S,' || r_object_id
  FROM dmi_dd_type_info_sp
 WHERE type_name ...;

SELECT 'apply,c,,REMOVE_DD_OBJECT,ID,S,' || r_object_id
  FROM dmi_dd_attr_info_sp
 WHERE type_name ...;

Fighting with Composer. Mac OS X

About a month ago I have bought MBP, and now I’m on a halfway of migrating from my preferred applications to something else (unfortunately, there is no replacement for TOAD (have you read latest news? DELL is about to acquire EMC) on Mac OS X). And on Friday I have faced with dumb problem: composer does not work on Mac OS X. Initially, I resolved this problem using following straightforward (that was my understanding about how eclipse works) recipe:

  • Download eclipse build for Mac OS composer is based on
  • copy Mac OS related files/directories (names contain macosx/cocoa) into composer’s plugins/directory
  • remove configuration/org.eclipse.osgi directory
  • actualize artifacts.xml and org.eclipse.equinox.simpleconfigurator/bundles.info files (replace win32/win64 entries by corresponding macosx/cococa entries from eclipse build for Mac OS)
  • profit:

After that I have found a more convenient way provided by EMC employee: Composer on Helios. Interesting, thoughts about incorrect composer distribution was raised five years ago:

Lastly, a small request from me. Before I left on my LOA and now that I am back I am always actively encouraging EMC to host and support Composer as a set of plugins from an update site, like the rest if the Eclipse community, as well as Composer as a product. But I am just one of a few voices saying this within EMC who obviously weigh our opinion up against the cost of hosting the update site. So if you would like to consume Composer from an update site, if you would like Composer to be an open -platform it is important that you let EMC know this by telling them directly or by commenting here.

but even in 2015 composer is distributed as a piece of dog crap.

Fighting with Composer

More than eight years ago EMC released the first version of “very promising” application for packaging and deploying Documentum artifacts – Documentum Composer, intended to kill Documentum Application Builder/Installer. Unfortunately, even after eight years this “promising application” is suitable neither for development nor for installation routines. Really, if you try find some early descriptions of Documentum Composer, for example: Introduction to Documentum Composer, – you will find that nothing has been changed since 6.0SP1 version, EMC bumps version numbers, but Composer still not able to do elementary things, for example, when I try to remove artifact from project I get a weird warning:

What the fuck is “other artifacts”, how can I find them? Open my preferred filemanager and try to find them manually? What a shame! How can I manually modify xml files? Edit via text editor (yeah, some folks think that if application uses XML to store data this means that application uses open format)? Bullshit! How can I edit this:

? It seems that EMC coders has never heard before about UTF-8 encoding and CDATA section in XML! Installation process is tedious as well, interesting, has anybody at EMC heard about java.io.tmpdir system property

or about file names in ZIP specification:

? I believe the answer is “no”. Ultimately, due to a lot of glitches in this “promising application” all experienced Documentum developers try to minimize the amount of interactions with Documentum Composer, the usage pattern varies from one developer to another, the most common patters are:

  • don’t use composer at all – developers write a set of API/DQL/dmbasic scripts and store them in VCS, this approach is extremely straightforward, but it typically requires two kinds sets: one for clean installation and another one for upgrading between application versions
  • use composer to transfer application between environments – developers perform changes in DEV environment using API/DQL/dmbasic scripts, after that they import Documentum artifacts into Composer

I prefer the second option because it is too boring to support two sets of scripts. But this approach has a couple of disadvantages:

  • I need to properly setup composer workspace on CI side, i.e. I need to unpack composer distribution, create empty workspace, create empty dummy project to force composer to create “DocumentumCoreProject” (yeah, Composer does not create DocumentumCoreProject automatically when you try to import project into empty workspace) and delete dummy project, after that the build scenario may look like (extremely error-prone!):
    • Delete composer project from workspace
    • Copy composer project from VCS to workspace
    • Register (import) composer project in workspace
    • Perform some updates in project (like replacing jar files)
    • Build project (or workspace)
  • Sometimes, depending on a target environment, it is required to set upgrade option (i.e. ignore, version or override) for certain artifacts – “manually” editing and committing default.dardef file is not an option

There is an interesting fact related to the first problem: composer does allow to import “external” projects into workspace:

but corresponding “emc.importProject” ant task does not – it requires project to be copied into workspace:

In order to resolve both problems mentioned above I developed a simple eclipse plugin, and now my ant scenario looks like:

<?xml version="1.0"?>
<project name="composer" default="all">

  <property file="${basedir}/src/main/resources/build.properties"/>

  <taskdef name="ap.importProject" classname="tel.panfilov.documentum.composer.ImportProjectAntTask"/>
  <taskdef name="ap.setUpgradeOption" classname="tel.panfilov.documentum.composer.SetUpgradeOptionAntTask"/>

  <target name="create-workspace" description="Create composer workspace">
    <ap.importProject project="<project name>" location="${composer.project.dir}"/>
    <ap.setUpgradeOption project="<project name>">
       <upgradeOptions>
          <artifact name="*" category="com.emc.ide.artifact.bpm.processContainer" value="IGNORE"/>
          <artifact name="wt_executing" category="com.emc.ide.artifact.bpm.processContainer" value="VERSION"/>
       </upgradeOptions>
    </ap.setUpgradeOption>
  </target>

  <target name="build-workspace" description="build eclipse project">
    <eclipse.incrementalBuild kind="full"/>
  </target>

  <target name="clean-workspace" description="clean eclipse project">
    <eclipse.incrementalBuild kind="clean"/>
  </target>

  <target name="all" depends="create-workspace, clean-workspace, build-workspace"/>

</project>

How to slow down ingestion

On Friday when I was trying to insert about 200 thousand objects into repository I have noticed that ingestion rate is much slower than I was expecting, after creating a couple of thread dumps I have found out that during instantiation of new object DFC performs extra fetches from repository:

These fetches are caused by definition of attribute default values in object type – every definition of default value causes extra fetch from repository:

Dumb dmbasic

Today I had faced with following “challenge”:

When dealing with composer I prefer to put all object types into one project and all TBOs into another, due to following reasons:

  • both content server and composer are extremely slow in installing object types – I have seen some situations when installing object types takes about two hours because content server performs millions database selects
  • TBO implementation changes more often than object type definition

but today I was need to attach default aspects to certain object types, in order to do that I had written a post-install procedure (and, yeap, my eyes had came out):

Const glabel As String          = "Label"
Const ginfo As String           = "Info"
Const gerror As String          = "Error"

Private Sub PrintMessage(mssg As String, mssgtype As String)
  If(mssgtype=glabel) Then
            Print "<BR><B><FONT size=3>"
            Print mssg
            print "</FONT></B>"
  ElseIf(mssgtype=ginfo) Then
            Print "<BR><FONT color=blue>"
            Print mssg
            print "</FONT>"
  ElseIf(mssgtype=gerror) Then
            Print "<BR><FONT color=red size=3>"
            Print mssg
            print "</FONT>"
  Else
            Print "<BR>" & mssg
  End If
End Sub

Private Sub AddAspect(TypeName2 As String, AspectName As String)
    Dim AddQryAPI As String
    Dim APIStatus As String
    Dim Status As Integer
    Dim QueryID As String
    Dim AttributeID As String

    AddQryAPI = "query,c,ALTER TYPE " & TypeName2 & " ADD DEFAULT ASPECTS " & AspectName
    Call PrintMessage("Adding the aspect " & AspectName & " to " & TypeName2, ginfo)
    QueryID = dmAPIGet(AddQryAPI)
    If QueryID <> "" Then
         Call PrintMessage("Adding success", ginfo)
         Status = dmAPIExec("close,c," & QueryID)
    Else
         msgStr$ = dmAPIGet("getmessage,c")
         Call PrintMessage(msgStr$, gerror)
         msgStr$ = "Failed to add default aspect " & AspectName
         Call PrintMessage(msgStr$, gerror)
         dmExit(1)
    End If
End Sub



Sub PostInstall(DocbaseName As String, UserName As String, Password As String)
  Dim SessionID As String

  SessionID= dmAPIGet("connect," & DocbaseName & "," & UserName & "," & Password)
  If SessionID ="" Then
    Print "Fail to connect to docbase " & DocbaseName &" as user " & UserName
    DmExit(-1)
  Else
    Print "Connect to docbase " & DocbaseName &" as user " & UserName
  End If

  Call AddAspect("type1", "aspect1")

End Sub

but when installing composer project I got weird error:

[FATAL]  An unexpected error has occurred during installation.

For more details, check the most recent error log located by default in 'C:\Temp\documentum\'.
com.emc.ide.installer.PostInstallException: Error running post-install procedure "postinstall"
        at internal.com.emc.ide.installer.DarInstaller.postInstall(DarInstaller.java:1574)
        at internal.com.emc.ide.installer.DarInstaller.doInstall(DarInstaller.java:669)
        at internal.com.emc.ide.installer.DarInstaller.doInstall(DarInstaller.java:334)
        at internal.com.emc.ide.installer.DarInstaller.doInstall(DarInstaller.java:303)
        at com.emc.ide.installer.popup.actions.InstallOperation.installDar(InstallOperation.ja
        at com.emc.ide.installer.popup.actions.InstallOperation.run(InstallOperation.java:80)
        at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:1
Caused by: com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerException: Failed to run dm
mbasic\dmbasic.exe -f G:\Users\andrey\work\vk\ord_int\dev\ContentServer\ComposerProject\esed_d
CKET=T0JKIE5VTEwgMAoxMwp2ZXJzaW9uIElOVCBTIDAKMwpmbGFncyBJTlQgUyAwCjEKc2VxdWVuY2VfbnVtIElOVCBTI
MTcwCmRvbWFpbiBJTlQgUyAwCjAKdXNlcl9uYW1lIFNUUklORyBTIDAKQSA3IGRtYWRtaW4KcGFzc3dvcmQgU1RSSU5HIF
ENPUWJVcURIbGI5MXhIRWE1bmUrNDUwWXo5RUZtUXQwTWZ0WkNCa0NnRlhuYThieWdsOW5qYz0KZG9jYmFzZV9uYW1lIFN
VydmVyX25hbWUgU1RSSU5HIFMgMApBIDggRENUTV9ERVYKc2lnbmF0dXJlX2xlbiBJTlQgUyAwCjExMgpzaWduYXR1cmUg
3MDRFWXBlaW5RNE1UOVVBZ1pCeGFVYkhRQ3B2R29PbFRJRlpVcDZ3WVZFUXMxZ3VYa1Nqai9YQ3lvaE04Tkk2bDFYCg==)
        at com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerUtils.executeCommand(Proced
        at com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerUtils.executeDmBasic(Proced
        at com.emc.ide.external.dfc.procedurerunner.ProcedureRunner.execute(ProcedureRunner.ja
        at internal.com.emc.ide.installer.DarInstaller.postInstall(DarInstaller.java:1570)
        ... 6 more
Caused by: java.io.IOException: Cannot run program "G:\app\emc\composer\7.2\plugins\com.emc.id
rror=740, The requested operation requires elevation
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)
        at com.emc.ide.external.dfc.procedurerunner.ProcedureRunnerUtils.executeCommand(Proced
        ... 9 more
Caused by: java.io.IOException: CreateProcess error=740, The requested operation requires elev
        at java.lang.ProcessImpl.create(Native Method)
        at java.lang.ProcessImpl.<init>(ProcessImpl.java:385)
        at java.lang.ProcessImpl.start(ProcessImpl.java:136)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
        ... 10 more

What the fuck did just happened here?

For some weird reason EMC coders decided that dmbasic executable requires administrative privileges (note shield icon – I bet dmbasic sends some information directly to EMC, otherwise I can’t find any reason why it requires administrative privileges):

How to fix that? Initially I was trying to find manifest editor in google, but after a couple of futile attempts I just edited dmbasic binary in HEX editor and got following (note extra spaces intended to preserve file size):

Weird Composer

Previously I put doubts on EMC SDLC process, yesterday I got another prove of that.

In Documentum 7 EMC increased length of user_name/group_name attribute:

and, as expected, failed to do that without bugs:

Now, let’s check the status of “innovation” in 7.2 release (2 years since Documentum 7):

Interesting, how is it possible to develop with Composer if it provides wrong information?