Silent UCF. HowTo

As was promised in previous post, I describe how make webtop customers happy.

What is the problem?

It seems that when Oracle acquired Sun it also acquired a bunch of security issues in JRE, for example, it’s pretty obvious that even if applet is signed by valid certificate it’s not enough to consider it as trusted – benign applet could be potentially used for malicious purposes (for example UCF applet is capable to launch any program on user’s machine, so attacker may exploit this capability). Security measures for this issue are obvious: just bind applet to specific website, and Oracle did it: JAR File Manifest Attributes for Security. Unfortunately these “innovations” are barely suitable for self-hosted applications – URLs vary from customer to customer, which makes it impossible to create applet suitable for every customer, though, I suppose EMC’s customers pay enough support fees to have an ability to request “custom” UCF applet from EMC, “custom” means that only manifest changes are required and such procedure could be easily automated – why do not create special service on customer portal? Instead of doing correct things EMC suggests something weird:

Actually, I double-checked Oracle’s documentation and was unable to find a place where they suggest to delegate security-related questions to business users, I suppose that EMC guys think that some posts in forums/blogs/etc have a status of official documentation, that, in fact, is not surprising: for example, IIG’s director of security prefers to get inspiration from this blog instead of performing code review:

Possible solutions

Anyway, Oracle’s suggestions related to security prompts are:

Though rule sets option is straightforward I do not think that it is helpful due to following reasons:

  • it costs money: “The JAR file must be signed with a valid certificate from a trusted certificate authority”
  • it is dangerous: “The Deployment Rule Set feature is optional and shall only be used internally in an organization with a controlled environment. If a JAR file that contains a rule set is distributed or made available publicly, then the certificate used to sign the rule set will be blacklisted and blocked in Java”, i.e. you risk to loose money spent for your certificate
  • if you already have a “valid certificate from a trusted certificate authority” why do not sign all applets in enterprise? So, this option is more suitable for cases when applets are embedded in hardware devices like KVMs, iLOs, etc and you unable to replace those applets
  • I think the best case for rule sets is enable certain known applets and block all other

Second option costs money too because it requires a “valid certificate from a trusted certificate authority”, but it’s the only disadvantage of this option. So, what did I do to disable all security prompts in previous video? Obviously, at first, we bought code-signing certificate.

Solution

Let’s take a look at security prompts raising when UCF applet gets loaded:

  • Java’s standard prompt asking whether user is going to trust applet’s certificate:
  • UCF’s prompt aimed to prevent malicious usage of applet (whitelisting):
  • Java’s prompt about not following security best practices:

For the third option EMC, as we already know, suggests to tick “Do not show this again for this app and web site” checkbox before accepting security warning. Besides the fact that delegating security-related questions to end-users is not a good idea, this options does not work properly. The problem is WDK generates special URLs for applets to prevent them from caching by JRE:

here “19gmgko28” and “98th” parts of URL are just encoded values of last modified time and size of applet file:

-bash-4.1$ stat -c "Last Modified: %Y, size: %s" \
> app/webapps/wp/wdk/system/ucfinit.jar
Last Modified: 1426684797, size: 304049
-bash-4.1$ groovysh
Groovy Shell (2.4.1, JVM: 1.6.0_45)
Type ':help' or ':h' for help.
groovy:000> import com.documentum.web.util.StringUtil
===> com.documentum.web.util.StringUtil
groovy:000> StringUtil.toUnsignedString(1426684797000,5)
===> 19gmgko28
groovy:000> StringUtil.toUnsignedString(304049,5)
===> 98th

Such behaviour of WDK has following disadvantages:

  • redeploying/updating webtop changes last modified time or/and size of applet file
  • in clustered environment it’s expected that applet file has different last modified time on different nodes

also we already know that the cornerstone of UCF troubleshooting is to delete everything related to Java and Documentum, reinstall JRE, press ctrl+alt+reset, etc – these activities obviously wipe user’s settings, it’s clear that ticking “Do not show this again for this app and web site” checkbox is just a temporary workaround, so, we decided to sign applet by our certificate. The procedure is straightforward:

  • edit META-INF/MANIFEST.MF, you should get something like (note Caller-Allowable-Codebase, Application-Library-Allowable-Codebase and Codebase headers):
    Manifest-Version: 1.0
    Built-By: dmadmin
    Application-Name: Documentum
    Created-By: 1.5.0_11-b03 (Sun Microsystems Inc.)
    Copyright: Documentum Inc. 2001, 2004
    Caller-Allowable-Codebase: docum-at-app
    Build-Date: January 17 2015 09:07 PM
    Ant-Version: Apache Ant 1.8.4
    Title: Documentum Client File Selector Applet
    Application-Library-Allowable-Codebase: docum-at-app
    Bundle-Version: 6.7.2220.0231
    Build-Version: 6.7.2220.0231
    Permissions: all-permissions
    Codebase: docum-at-app
    
  • remove old signature files, i.e. META-INF/DOCUMENT.RSA and META-INF/DOCUMENT.SF
  • run jarsigner to sign applet – do not forget to specify TSA URL

after that applet is ready for deployment in production. What about another two security prompts? I suppose it is obvious that whitelisting capability is now useless because now JRE performs the same checks, below is an example of security notice when applet gets loaded from non-trusted URL:

because of these considerations, I decided to disable whitelisting capability, unfortunately the only option to disable it in UCF is decompile com.documentum.ucf.client.install.installer.security.impl.Whitelist class: methods isHostAllowed(String) and isHostAllowed(String, IHostVerifier) must always return true (I bet it doesn’t worth to mention that after replacing class in applet you must sign applet again).

And finally, to remove the first security prompt I read Deployment Configuration File and Properties article and decided that manipulating by User Level configuration files is not a good idea, but System Level configuration files could be easily distributed among users’ machines in enterprise using various deployment tools, so, I did following (note naming convention of certificate alias):

C:\Windows\system32>md C:\Windows\Sun\Java\Deployment

C:\Windows\system32>cd C:\Windows\Sun\Java\Deployment

C:\Windows\Sun\Java\Deployment>type CON > deployment.config
deployment.system.config=file\:C\:/Windows/Sun/Java/Deployment/deployment.properties

C:\Windows\Sun\Java\Deployment>type CON > deployment.properties
deployment.system.security.trusted.certs=C\:\\Windows\\Sun\\Java\\Deployment\\trusted.certs

C:\Windows\Sun\Java\Deployment>keytool.exe -importcert -keystore trusted.certs -file G:\Users\andrey\work\cert.crt -alias deploymentusercert$tsflag$loc=http//docum-at-app:8280##docbase:http//docum-at-app:8280##from:http//docum-at-app:8280
Enter keystore password:
Re-enter new password:

...

Trust this certificate? [no]:  yes
Certificate was added to keystore

Voilà.

Bulk fetches. GC competition

For my load profile I got following results:

Java 6 (CMS wins, ParallelOld looses):

Java 7 (G1 wins, ParallelOld looses):

Interestingly, flushing object from cache after “fetch”, i.e.

if (_iterator.hasNext()) {
    IDfPersistentObject object = _iterator.next();
    try {
        _session.flushObject(object.getObjectId());
    } catch (DfException ex) {
        throw new RuntimeException(ex);
    }
    return;
}

improve results by 20% percent:

Because my load profile is not typical for any DFC application, I decided to perform same benchmarks for regular fetches, i.e. session.getObject() – I suppose such load is typical for BPM, DFS and REST, unfortunately, content server is so slow in performing sysobject fetches that it is hardly possible to notice a significant difference between garbage collectors:

So, I decided to switch to non-sysobject objects. Stay tuned.

GC challenge

As I already mentioned in previous post I decided to perform microbenchmarks to figure out what GC settings are optimal for customer’s application. Unfortunately, I had faced with an unexpected problem: JDK documentation is very obscure in questions related to GC settings, for example, Java HotSpot VM Options states following:

it seems to be wrong because “-” (minus sign) before option means disabling corresponding option, but not enabling it 😦 Fortunately, I discovered two useful JVM options: -XX:+PrintFlagsFinal and -XX:+PrintCommandLineFlags, which allow to get information about enabled options and resulting commandline arguments, for example:

]$ java -XX:-UseParallelOldGC -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal \
> -version | grep "UseParallelOldGC\|UseParallelGC"
-XX:InitialHeapSize=67108864 -XX:MaxHeapSize=1073741824 
   -XX:ParallelGCThreads=2 -XX:+PrintCommandLineFlags 
   -XX:+PrintFlagsFinal -XX:+UseParallelGC -XX:-UseParallelOldGC
     bool UseParallelGC                            := true            {product}
     bool UseParallelOldGC                         := false           {product}
java version "1.6.0_27-rev"
Java(TM) SE Runtime Environment (build 1.6.0_27-rev-b21)
Java HotSpot(TM) Server VM (build 20.2-b06, mixed mode)

i.e. -XX:-UseParallelOldGC option disables parallel garbage collection for the full collections and parallel garbage collection for scavenges is enabled by default for my JVM. Also I found a nice blogpost which clarified a lot of my doubts. I slightly redrew the diagram provided in that post:

because connection between CMS and MSC GCs seems to be confusing, and now it’s clear that JVM supports seven combinations of GCs. For Java 6 these combinations are:

Short options Long options Description
-XX:+UseG1GC -XX:+UseG1GC Use G1 collector for young and tenured generations
-XX:+Parallel -XX:+Parallel -XX:-UseParallelOldGC Use Parallel Scavenge collector for young generation, and Serial for tenured
-XX:+UseConcMarkSweepGC -XX:+UseConcMarkSweepGC -XX:+UseParNewGC Use ParNew collector for young generation and CMS for tenured
-XX:+UseConcMarkSweepGC -XX:-UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseParNewGC Use Serial collector for young generation and CMS for tenured
-XX:+UseSerialGC -XX:+UseSerialGC Use Serial collector for young generation and Serial for tenured
-XX:+UseParNewGC -XX:+UseParNewGC Use ParNew collector for young generation and Serial for tenured
-XX:+UseParallelOldGC -XX:+Parallel -XX:+UseParallelOldGC Use Parallel Scavenge collector for young generation, and Parallel for tenured

In Java 7 -XX:+UseParallelOldGC option is enabled by default, so GC matrix for Java 7 differs by two rows.

Bulk fetches

In September 2014 we got a new customer which had a EDMS system with a whole bunch of performance issues, the system is based on heavy customization of Webtop. The main problem was that application was performing a lot of fetches from docbase, for example, in some cases opening document properties page was accompanied by 10000 fetches (we already know that fetches are slow), so, users’ experience was predictable. To eliminate the most of bottlenecks caused by such weird design we had performed some optimizations intended to decrease the number of fetches – some things got replaced by DQL queries, another things got cached (here the technique described in Power Pivot had helped us a lot – that was pretty easy to find slow components and components with skewed data). Unfortunately, due to application’s design it was not possible to completely eliminate all unnecessary fetches – it was looking the same as completely rewrite application. So, I decided to implement “bulk fetches”, i.e. try to load objects, having the same type, by a single RPC command.

Why are fetches slow?

At first, I think it is worth to shed light on IDfSession#getObject and IDfPersistentObject#fetch methods. Documentation says following about these methods:

getObject

IDfPersistentObject getObject(IDfId objectId) throws DfException

Returns a Documentum server object.

fetch

boolean fetch(String typeNameIgnored) throws DfException

Fetches this object from the repository without placing a lock on the object.
Use this method to ensure that you are working with the most recent version of the object. You must have at least BROWSE permission on an object to call the fetch method.
Without a lock, there is no guarantee that you will be able to save any changes you make to the object since another user may checkout the object while you have it fetched. If you fetch an object, you cannot use the checkin method to write the object back to the repository. You must use the save method.

I have no idea who in EMC writes misleading documentation, but the only correct description for these methods is: “both methods retrieve current object’s snapshot from repository if corresponding DFC object is not dirty“. To clarify last statement let’s perform some tests based on technique described in RPC Commands (draft):

public class Test {

    public static void main(String[] args) throws DfException {
        IDfSession session = new DfClientX().getLocalClient().newSession(
                "ssc_dev", new DfLoginInfo("dmadmin", "dmadmin"));
        System.out.println(session.getSessionConfig().dump());
        // cache is empty
        log(session, "FIRST GET OBJECT START");
        doTrace(session, true);
        session.getObject(new DfId("0901ffd7805b03d8"));
        doTrace(session, false);
        log(session, "FIRST GET OBJECT END");
        // cache already contains object
        log(session, "SECOND GET OBJECT START");
        doTrace(session, true);
        IDfPersistentObject object = session.getObject(new DfId(
                "0901ffd7805b03d8"));
        doTrace(session, false);
        log(session, "SECOND GET OBJECT END");
        // dirty object
        object.setString(DfDocbaseConstants.OBJECT_NAME, "");
        log(session, "THIRD GET OBJECT START");
        doTrace(session, true);
        session.getObject(new DfId("0901ffd7805b03d8"));
        log(session, "THIRD GET OBJECT END");
        doTrace(session, false);
        session.disconnect();
    }

    private static void log(IDfSession session, String message)
        throws DfException {
        session.apply(null, "STAMP_TRACE",
                new DfList(new String[] {"MESSAGE" }), new DfList(
                        new String[] {"S" }), new DfList(
                        new String[] {message }));
    }

    private static void doTrace(IDfSession session, boolean enable)
        throws DfException {
        if (enable) {
            session.apply(null, "LOG_ON", new DfList(new String[] {"DETAIL" }),
                    new DfList(new String[] {"B" }), new DfList(
                            new String[] {"T" }));
        } else {
            session.apply(null, "LOG_OFF", new DfList(new String[] {}),
                    new DfList(new String[] {}), new DfList(new String[] {}));
        }

    }

}

Session log is:

// cache is empty
FIRST GET OBJECT START
[DM_SESSION_I_OPERATION_STARTED]info:  "Operation SysObjFullFetch started."
[DM_SESSION_I_OPERATION_ARGUMENTS]info:  "
Object Id: 0901ffd7805b03d8
Argument Object:
2
OBJ NULL 0 0 0 0
2
FOR_REVERT BOOL S 0
F
CACHE_VSTAMP INT S 0
174476
"
[DM_SESSION_I_OPERATION_ENDED]info:  "Operation SysObjFullFetch completed."
FIRST GET OBJECT END
// cache already contains object
SECOND GET OBJECT START
[DM_SESSION_I_OPERATION_STARTED]info:  "Operation IsCurrent started."
[DM_SESSION_I_OPERATION_ARGUMENTS]info:  "
Object Id: 0901ffd7805b03d8
Argument Object:
2
OBJ NULL 0 0 0 0
4
OBJECT_TYPE STRING S 0
A 11 dm_document
i_vstamp INT S 0
0
CACHE_VSTAMP INT S 0
174476
TYPE_CACHE_VSTAMP INT S 0
4707
"
[DM_SESSION_I_OPERATION_ENDED]info:  "Operation IsCurrent completed."
SECOND GET OBJECT END
// dirty object
THIRD GET OBJECT START
THIRD GET OBJECT END

I.e, for uncached object DFC sends fetch (SysObjFullFetch for sysobjects) RPC-command, for cached object DFC sends IsCurrent RPC-command, and for dirty object DFC sends nothing, the behaviour of IDfPersistentObject#fetch method is the same: DFC tries to figure out whether object’s snapshot is actual or not by sending IsCurrent RPC-command only if object is not dirty, if IsCurrent RPC command returns false DFC sends fetch RPC. Note, if your coder writes something like:

IDfPersistentObject object = session.getObject("bla-bla-bla");
object.fetch(null);

he is an idiot – object.fetch() reiterates the same logic which already was performed by session.getObject(). So in best case IDfSession#getObject sends only one RPC command (IsCurrent or fetch), in worst case (object is present in DFC cache but stale) it sends two RPC commands (IsCurrent and fetch). In my case application performs a lot of fetches, hence cache hit tends to be zero, moreover my situation is complicated by the fact that SysObjFullFetch RPC command is extremely slow because it submits five SQL queries to database (IsCurrent submits only one SQL query), those queries are:

-- determine object type by r_object_id
SELECT dm_dbalias_B.I_TYPE
  FROM DMI_OBJECT_TYPE dm_dbalias_B
 WHERE dm_dbalias_B.R_OBJECT_ID = :dmb_handle

-- retrieve data from database
  SELECT *
    FROM TYPE_RV dm_dbalias_B, TYPE_SV dm_dbalias_C
   WHERE (    dm_dbalias_C.R_OBJECT_ID = :dmb_handle
          AND dm_dbalias_C.R_OBJECT_ID = dm_dbalias_B.R_OBJECT_ID)
ORDER BY dm_dbalias_B.R_OBJECT_ID, dm_dbalias_B.I_POSITION

-- get identifier of object's ACL (totally useless)
SELECT dm_dbalias_B.R_OBJECT_ID
  FROM DM_ACL_S dm_dbalias_B
 WHERE (dm_dbalias_B.OWNER_NAME = :p00 
    AND dm_dbalias_B.OBJECT_NAME = :p01)

-- retrieve object's ACL
SELECT *
    FROM DM_ACL_RV dm_dbalias_B, DM_ACL_SV dm_dbalias_C
   WHERE (    dm_dbalias_C.R_OBJECT_ID = :dmb_handle
          AND dm_dbalias_C.R_OBJECT_ID = dm_dbalias_B.R_OBJECT_ID)
ORDER BY dm_dbalias_B.R_OBJECT_ID, dm_dbalias_B.I_POSITION

-- check whether ACL is actual or not (totally useless)
SELECT dm_dbalias_B.R_OBJECT_ID
  FROM DM_ACL_S dm_dbalias_B
 WHERE (    dm_dbalias_B.R_OBJECT_ID = :dmb_objectp
        AND dm_dbalias_B.I_VSTAMP = :dmb_versionp)

Bulk fetches

Surprisingly, but DFC has a method which allows to construct persistent objects using DQL query:

getObjectsByQuery

IDfEnumeration getObjectsByQuery(String dql,
                                 String optionalTypeName)
                                 throws DfException

Returns an enumeration of persistent objects using data obtained through a query.
The query doesn’t need to contain all the object attributes but there are a minimum subset of attributes that must be present. All queries must include r_object_id and i_vstamp attribute. The query must include r_object_type, r_aspect_name, i_is_replica and i_is_reference if the object has these attributes. The optionalTypeName is required if the object does not have the attribute r_object_type, eg: dm_user. When a query is done on dmi_queue_item, the attribute “source_docbase” must be included in the query.
Note that the objects returned by this call may not be fully populated with data. Only data present in the query will exist in the returned objects. As long as your subsequent access to the objects only reference the populated data then no “fetch” of the full object will occur. As soon as you try to access object data that is not present then an internal “fetch” will be triggered to obtain the remaining data. This process is transparent to the object user.

Unfortunately, IDfSession#getObjectsByQuery method didn’t feet my needs due to following reasons:

  • it returns deprecated Enumeration, backed up by ArrayList – large resultsets consume a lot of memory
  • it was required to rewrite a lot of DQLs to inject object’s attibutes – application was initially designed to retrieve a set of objects’ identifiers using DQL query and then perform session.getObject() to get a set of persistent objects

so, I started seeking for a way to perform “bulk fetches” having a set of objects’ identifiers. After some investigation I discovered an option to construct persistent objects from IDfCollection:

PersistentObjectManager objectManager = ((ISession) session)
        .getObjectManager();
IDfCollection collection = ...;
while (collection.next()) {
    IDfPersistentObject object = objectManager
            .getObjectFromQueryRow(collection, objectType);
}

Now, the problem has been narrowed to constructing IDfCollection object with required attributes. I have proposed following options:

  • DQL_MATCH RPC command:
    StringBuilder builder = new StringBuilder();
    builder.append(DfDocbaseConstants.R_OBJECT_ID).append(" IN (");
    for (Iterator<String> iter = ids.iterator(); iter.hasNext();) {
        builder.append('\'').append(iter.next()).append('\'');
        if (iter.hasNext()) {
            builder.append(',');
        }
    }
    builder.append(")");
    PersistentObjectManager manager = ((ISession) session)
            .getObjectManager();
    IDfCollection collection = session.apply(null, "DQL_MATCH", new DfList(
            new String[] {"QUERY_TYPE", "QUERY_PREDICATE" }), new DfList(
            new String[] {"S", "S" }), new DfList(new String[] {
        "dm_document", builder.toString() }));
    while (collection.next()) {
        IDfPersistentObject object = manager.getObjectFromQueryRow(
                collection, "dm_document");
    }
    
  • Regular DQL query with a trick to save ordering of repeating attributes (complete code could be found on github):
    IDfType type = session.getType("dm_document");
    boolean hasRepeatings = false;
    StringBuilder builder = new StringBuilder("SELECT ");
    for (int i = 0, n = type.getTypeAttrCount(); i < n; i++) {
        IDfAttr attr = type.getTypeAttr(i);
        if (attr.isRepeating()) {
            hasRepeatings = true;
        }
        builder.append(attr.getName());
        builder.append(",");
    }
    builder.append(DfDocbaseConstants.R_OBJECT_ID).append(",");
    builder.append(DfDocbaseConstants.I_VSTAMP);
    if (hasRepeatings) {
        builder.append(',');
        builder.append("i_position");
    }
    builder.append(" WHERE ").append(DfDocbaseConstants.R_OBJECT_ID)
            .append(" IN (");
    for (Iterator<String> iter = ids.iterator(); iter.hasNext();) {
        builder.append('\'').append(iter.next()).append('\'');
        if (iter.hasNext()) {
            builder.append(',');
        }
    }
    builder.append(")");
    builder.append(" ORDER BY ").append(DfDocbaseConstants.R_OBJECT_ID);
    if (hasRepeatings) {
        builder.append("i_position  DESC");
    }
    PersistentObjectManager manager = ((ISession) session)
            .getObjectManager();
    IDfCollection collection = new DfQuery(builder.toString()).execute(
            session, IDfQuery.DF_EXEC_QUERY);
    while (collection.next()) {
        IDfPersistentObject object = manager.getObjectFromQueryRow(
                collection, "dm_document");
    }
    
    

Later, DQL_MATCH demonstrated bad performance – it has only three times performance improvement over regular fetches:

moreover sometimes it does not work properly, so DQL_MATCH was not an option anymore.

It seemed that the last challenge was to identify the optimal count of objects’ identifies in “IN ()” clause – I already knew that DQL queries have length restrictions and in general case performance depends on the length of query and after some microbenchmarks I have found that the optimal value is somewhere between 500 and 100:

But I have noticed a flat region on graphs which is cased by garbage collection activity and now the challenge is to find optimal GC settings.

Another one security “best practice” from EMC

If something goes wrong – turn off DEP:

What is DEP? I have found a nice explanation here: How do ASLR and DEP work? And The Wikipedia has following opinion about DEP:

DEP occasionally encounters software problems, usually with older software that was not compiled and tested to conform to its restrictions. Users have experienced problems using various command-line commands that are a part of Microsoft’s Services for Unix, which is included as part of Vista as well as Windows Server 2003 R2.

These problems may be prevented by disabling DEP, but this increases vulnerability of the system to malware. DEP can be turned off on a per-application basis, or turned off entirely for all non-essential Windows programs and services. Microsoft recommends that DEP not be globally disabled where an application malfunctions due to incompatibility with DEP. Instead, the supplier of the offending software should be contacted for an updated version that does not violate DEP; until the problem is corrected DEP may be disabled on an exception basis for the offending application only.

DEP is applied to an entire process, so even an application compatible with DEP may need to have it disabled if a non-DEP-compliant extension is added that runs in the same process space. For example, DEP-related problems can occasionally occur with DEP-compliant core operating system components such as Windows Explorer, Internet Explorer and Windows Installer as they support in-process third party extensions or plugins that may not be DEP-compliant