To sell or not to sell…

It has been 11 months since I posted my last blogpost about vulnerabilities in Documentum stack, actually, I didn’t stop researching (it is interesting, and flatters my vanity) – I just stopped posting due to following two reasons:

  • There are “gifted” employees in EMC, this employees do think they are experts in bot security and Documentum and periodically (or day by day 🙂 ) read my blog and fecklessly try to understand what is written here and somehow remediate security flaws – such attempts are doomed to failure
  • Doing the same more officially, like file vulnerability reports to CERT/CC, brings a lot of headache – I consider vulnerability researching as a hobby, so, I have no interest to participate in such dumb activities – I tried and wasn’t satisfied with the results

Moreover, I have found that this activity improves neither product nor customer experience – D2 perfectly demonstrates this point. By the way, during last 11 months I discovered about 30 vulnerabilities in Documentum products and I periodically receive e-mails like:

Good Day, Andrey. My name is Roman, I found you contacts through seclists.org, where your HTTP session poisoning in EMC Documentum WDK-based applications causes arbitrary code execution and privilege elevation vulnerability was published.
I would like to offer you a collaboration that could be beneficial for both of us. I`m purchasing 0day exploits and vulnerabilities in software, big websites, routers. Would you be interested to sell it?
Looking forward to your reply.

Regards,
Roman.

What to do?

Copying version tree. Part II

Wow, Marian Caikovski have found a bug in my old blogpost about copying version tree 🙂 Though it is not clear what “The described method will work even if the root version in the source object has been previously deleted. But the copies will not be entirely valid.” does mean, because I observe the following behaviour:

Connected to Documentum Server running Release 7.3.0000.0214  Linux64.Postgres
Session id is s0
API> create,c,dm_document
...
0902987880002cf1
API> save,c,l
...
OK
API> checkout,c,l
...
0902987880002cf1
API> checkin,c,l
...
0902987880002cfc
API> fetch,c,0902987880002cfc
...
OK
API> checkout,c,l
...
0902987880002cfc
API> checkin,c,l
...
0902987880002cfe
API> destroy,c,0902987880002cf1
...
OK
API> fetch,c,0902987880002cf1
...
OK
API> saveasnew,c,l
...
[DM_SYSOBJECT_E_IS_DELETED]error:  
  "The sysobject '' has been deleted and can not be updated/deleted."

which could be fixed using following trick:

API> set,c,l,i_is_deleted
SET> F   
...
OK
API> saveasnew,c,l
...
0902987880002cf3

The complete code is available on github.

ACL performance

This blogpost is a follow-up for WTF??? blogpost, but describes the similar problem from Content Server perspective.

Two months ago one large company asked me to help them with performance issue, that time I had written two blogposts about how to diagnose performance issues in Documentum environment: Diagnostics challenge. Part I and Diagnostics challenge. Part II, but I did’t shed a light on what was the root cause. Interesting thing here is a fact that that time I was already familial with such performance problem in Documentum – we had filed a bug to EMC about four years ago but it is still not yet resolved: initially EMC was promising to resolve this performance problem in upcoming 7.x release, later they said something like “it requires a lot of effort – we are not going to fix it”.

Well, I’m not sure about all existing ECM solutions, but can definitely say about Documentum: Documentum (as a product) is not a ECM solution because it lacks concept of business roles (i.e. when business-users gets some capabilities according to the current context), and, hence, it is not suitable for enterprises. To prove my point let’s examine the most basic capability: certain user is able to read certain document. And the question is under what circumstances user gets read access to document? Actually, there are a plenty of reasons, some of them are:

  1. user is somehow involved in a business process related to this document, i.e. our user is an author, or reviewer, or addressee, or somebody else
  2. document was somehow classified and user gets access due to this classification, for example, members of legal department have access to all legal documents
  3. user is a legal successor of user from #1
  4. user is a secretary/assistant of user from #1
  5. user is a supervisor of user from #1
  6. user is a big boss and wants to see all documents in enterprise
  7. user is not a big boss, but wants to see all documents in branch office
  8. document somehow relates to another document, accessible by our user

I do not pretend that the list above is complete, but I do believe that it is common for all enterprises, and the problem is that Documentum (as a product) does not cover these cases – even combining #1 and #2 is already a challenge (check What is wrong in Documentum. Part III blogpost); #8 requires functionality of dynamic groups (check Dynamic groups. Advances. Part IV), which is not properly documented; for #3, #4 and #5 the best approach seems to be do not use direct grants to users, but instead create an associated group for each user – all these tricks require additional coding, however EMC thinks that companies do not buy their “product” because of costs of database licenses, but not because their “product” doesn’t fit customers’ needs. LOL 🙂

So, under such circumstances every customer/developer tends to invent own square wheel, and in most cases my job is to understand what was done and somehow improve that. The problem was following: customer’s application, every time when document was being saved to repository, was recalculating access permissions and was updating related acl as well – at first glance such implementation looks reasonable, but Content Server behaviour is extremely weird in such case: if you are saving acl which actually hasn’t been changed it takes extremely long time:

-- list of ACLs which contain more than 10000 accessors
API> ?,c,select r_object_id from dm_acl group by r_object_id 
     having count(r_accessor_name)>10000 enable(row_based)
r_object_id     
----------------
45024be98002b10c
45024be98002a74d
45024be98002a645
45024be98002a74c
(4 rows affected)

API> fetch,c,45024be98002a74d
...
OK

API> save,c,45024be98002a74d
... <--- 30 seconds 
OK

API> save,c,45024be98002a74d
... <--- double check: 30 seconds again
OK

--
-- now Content Server magic: we are adding
-- new record to acl
--
API> grant,c,45024be98002a74d,dm_read_all,AccessPermit,,6
...
OK

API> save,c,45024be98002a74d
... <--- less than a second
OK

Let’s investigate the difference between two cases (saving acl as is and adding new record).

First case:

[DM_SESSION_I_SESSION_START]info:  "Session 01024be98000c241 started for user dmadmin."


Connected to Documentum Server running Release 7.2.0030.0195  Linux64.Oracle
Session id is s0
API> apply,c,,SQL_TRACE,SESSION_ID,S,01024be98000c241,LEVEL,I,10
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...
USER ATTRIBUTES

  result                          : T

SYSTEM ATTRIBUTES


APPLICATION ATTRIBUTES


INTERNAL ATTRIBUTES


API> save,c,45024be98002a74d
...
OK
API> Bye
[dmadmin@docu72dev01 ord-dars]$ grep -i select \
> /u01/documentum/cs/dba/log/00024be9/dmadmin/01024be98000c241 | wc
  10800  173071 1811563

Second case:

[DM_SESSION_I_SESSION_START]info:  "Session 01024be98000c246 started for user dmadmin."


Connected to Documentum Server running Release 7.2.0030.0195  Linux64.Oracle
Session id is s0
API> apply,c,,SQL_TRACE,SESSION_ID,S,01024be98000c246,LEVEL,I,10
...
q0
API> next,c,q0
...
OK
API> dump,c,q0
...                                
USER ATTRIBUTES

  result                          : T

SYSTEM ATTRIBUTES


APPLICATION ATTRIBUTES


INTERNAL ATTRIBUTES


API> grant,c,45024be98002a74d,dm_browse_all,AccessPermit,,3
...
OK
API> save,c,45024be98002a74d
...
OK
API> Bye
[dmadmin@docu72dev01 ord-dars]$ grep -i select \
> /u01/documentum/cs/dba/log/00024be9/dmadmin/01024be98000c246 | wc
     28     719    9178

Wow, 10800 select statements in first case and just 28 select statements in second case! Looks like something is wrong, doesn’t it? Fortunately, four years ago I was dealing with another ACL performance issue and, in order to prove their wrong opinion, EMC had shared source code related to ACL processing, below is a code snippet which demonstrates wrong behaviour:

  // In order to validate the ACE's for this ACL, we will
  // find the first ACE that has been modified (r_accessor_name,
  // r_accessor_permit, r_accessor_xpermit, permit_type and
  // application_permit) and then validate all ACE's from that
  // entry to the end of the ACE list.
  int from = 0;
  if (changed[_pos._accessorName] > 0)
    from = changed[_pos._accessorName];
  if (changed[_pos._accessorPermit] > 0)
    from = min(from, changed[_pos._accessorPermit]);
  if (changed[_pos._accessorXPermit] > 0)
    from = min(from, changed[_pos._accessorXPermit]);
  if (changed[_pos._permitType] > 0)
    from = min(from, changed[_pos._permitType]);
  if (changed[_pos._applicationPermit] > 0)
    from = min(from, changed[_pos._applicationPermit]);

Do you see a mistake? Here it is:

  int from = 0;
  // from is always 0 if we haven't changed accessors
  if (changed[_pos._accessorName] > 0)
    from = changed[_pos._accessorName];

and the correct code is:

  int from = this->GetAttrValueCount(_pos._accessorName);
  if (changed[_pos._accessorName] > 0)
    from = changed[_pos._accessorName];

So, just one line to fix an issue.

What is the date today?

groovy:000> cal = Calendar.getInstance();
===> java.util.GregorianCalendar
groovy:000> cal.set(2016,Calendar.SEPTEMBER,12);
===> null
groovy:000> cal.add(Calendar.DAY_OF_YEAR, 120);
===> null
groovy:000> cal.getTime();
===> Tue Jan 10 01:23:10 AEDT 2017

May be OpenText guys have changed their mind??

UPD.

There is a hypothesis about January 13:

ETL 01: Documentum -> MuleSoft -> Filesystem

Interesting blogpost about MuleSoft and Documentum, though I prefer to use Apache Camel 🙂

paulsundquist

This post is the first in a series focused on Extract-Transform-Load tools and techniques that I will discuss on this blog.

MuleSoft is an excellent tool to integrate real time updates (such as approved documents) from one system into another system.  MuleSoft has a very rich developer community with lots of examples, a good YouTube channel, and training …. I recommend this free 8 week course to learn the fundamentals.

We can use the Documentum REST interface to run a Documentum query, then either store the results or extract related PDF files from the system.

You can image the use cases for this set of tools:

  • Migrate content into or out of Documentum
  • Synchronize master data between systems
  • Publish content from Documentum to Box, Dropbox, Microsoft SkyDrive, Google Drive, etc.

Postman

Anyone new to the Documentum REST will want to use Postman to investigate some of the URLs that…

View original post 366 more words

Docker challenge

This blogpost is a follow-up for Think soberly. Is it hard? blogpost – that time it was unclear what ECM was going to do with docker (though, I was suspecting that nothing good would happen), so there was nothing to discuss, now EMC has released something and we are able to discuss pros and cons of their “solution”. It is also worth noting following blogposts related to impartial user experience:

There are also two interesting things:

  1. All this Docker stuff somehow relates to DevOps activities, and I used to be a DevOps from 2005 to 2008 – long before it became a mainstream 🙂
  2. Linux Containers are not something new or extraordinary – I already mentioned that in Docker and Documentum. Part I blogpost – just a couple of guys decided to promote this concept to a mainstream and created OOTB solution

So, why containers?

There is only one correct answer for this question: we have a lot of similar configuration items and want to automate their maintenance routines, moreover, we do think that maintenance overhead introduced by containers won’t exceed overheads introduced by other techniques. The problem is there is no clear boundary when containers are good and when they are harm, but it is important to understand that containerized environment is not a goal – it is an instrument. Let’s elaborate this point.

Imagine we are a hosting provider and we host a lot of php (potentially insecure) websites, and our goals are:

  • provide resources to customers according to their plan
  • provide enough flexibility to customers, e.g. customers are able to configure http-server
  • isolate customers from each other
  • do not use virtualisation due to overheads

Is it a Docker case? For sure yes!

Now, imagine that we are a software development company and our goal is to automate SDLC routines, for example, we want to test our product against different versions (and implementations) of third-party software, moreover, we want to test both releases and patches/hotfixes.

Is it a Docker case? For sure yes! Moreover, I pursue this case only.

Again, imagine that we are a software development company and our goal is to provide demo-version of our software for potential customers, i.e. we are interested that any potential customer might be able to evaluate our software immediately without reading tons of useless manuals.

Is it a Docker case? I’m not sure – I would prefer to build virtual machine for sober customers and docker image for geeks – just to follow a mainstream 🙂

Now, imagine that we are a customer, and have spent last money for Documentum, we have production environment and, may be, a test/uat environment.

Is it a Docker case? For sure not!

Now, about official Documentum Docker images

What EMC is doing is nailing screws using hammer 🙂 They understand neither applicability of Docker, nor the technology. Let’s provide some examples.

Below is a part of Alvaro’s blogpost:

Note, that it does not fit to a one screen and seems to be incomplete and wrong, due to following considerations:

  • “Docker image with Documentum Administrator” sounds completely ridiculous – apache tomcat perfectly fits “containerization concept” and the only correct way to run java-application inside Docker container is:
    [root@rhel73docker01 ~]# docker run -d -h webtop68dev01 \
    > -v /u01/distr/webtop-6.8-0000.0286.war:/usr/local/tomcat/webapps/webtop.war:ro \
    > tomcat:8.0
    3ba6af536629112fcc006ad68e280222c44b1ed73e2be34d7379f04e3f753c76
    [root@rhel73docker01 ~]# 
    

    actually, it is also required to map log directory and directories, containing dfc.properties and dfc.keystore files (dfc.config.dir and dfc.data.dir), but the whole idea, I believe, is absolutely clear – just one shell command to run a container

  • it is absolutely not clear how to manage this Docker image in distributed environment: EMC has written some bootstrap procedure, where I need to setup DOCBROKER_IP and DOCBROKER_PORT parameters, what am I supposed to do when I have multiple content servers? It is absolutely clear that I should place the contents of dfc.config.dir and dfc.data.dir outside of container (surprisingly, EMC implemented that in case of D2, though the values of dfc.data.dir and dfc.tokenstorage.dir parameters in Alvaro’s blogpost seem to be wrong)

As regards to Content Server, there is nothing to discuss thoroughly – their scripts are a mess. The only thing I want to discuss in this context is an applicability of Docker containers for Content Server.

I do believe that new technologies must accumulate previously received knowledge and best practices, for example, in case of Content Server I do think that the best storage option is a NAS appliance with 1-2Tb volumes, unfortunately, it seems that Docker engine experiences difficulties with such setup:

Summarising two previous issues it gets clear that if I want to implement my best practices I need manage NFS-volumes inside Docker container(s), yes, it is possible, but requires to run extra services inside container (rpcbind, portmapper, rpc.statd and rpc.lockd), which, no doubts, complicates my environment. But the problem here is not a fact that I have found an extremely specific case which does not fit Docker environment well (I can provide more), the problem is that EMC didn’t find/describe it – all what we have is a set of smelling shell scripts which do “something” and customers are supposed to guess what those scripts do and what will work and what will not.