Loading presentation...

Present Remotely

Send the link below via email or IM


Present to your audience

Start remote presentation

  • Invited audience members will follow you as you navigate and present
  • People invited to a presentation do not need a Prezi account
  • This link expires 10 minutes after you close the presentation
  • A maximum of 30 users can follow your presentation
  • Learn more about this feature in our knowledge base article

Do you really want to delete this prezi?

Neither you, nor the coeditors you shared it with will be able to recover it again.


CQ5.5 Lessons Learned

No description

David Beaudet

on 28 March 2013

Comments (0)

Please log in to add your comment.

Report abuse

Transcript of CQ5.5 Lessons Learned

Use client libraries to consolidate and minify your CSS and JavaScript. Separate your JS (under /apps/appname/component path) and your CSS (under /etc/designs/appname) and use categories and embeds (and possibly dependencies) to roll up all of your client libraries into a few consolidated files at publish time. Also, separate your publish components from your author components by checking the WCM mode before including client libs. Otherwise, you might end up (like we did) finding a 7.3MB widgets.js file included for no reason on your home page.
Remember that /etc/designs/yourapp/jcr:content contains the list of components permitted for a given template – you can either store it in SVN or let it live on your production authoring instance – but you should choose one approach over the other and adapt build scripts accordingly.
Use web browser tools and free on-line web site performance analysis tools to suggest ways of improving the load times of your pages and minimizing the number of requests.
Consolidated Search: we had a requirement to spider and show results from non-CQ5 managed web sites within our general site search so we wrote a spider that stores the content of select pages from those sites into the JCR. The web site source is used as a facet so users can opt to select / deselect based on the source web site. SOLR (or another external search engine) could be used for this purpose as well and CQ5 templates would simply wrap the SOLR restful APIs to render results.
Use latest dispatcher for benefit of native HTTPS proxying
If at first Adobe support isn’t helpful, be persistent, request a phone call – the quality of response is highly variable – I’ve also heard that specifying “Sling” as the component leads to the best customer support personnel, but I haven’t confirmed that.
Avoid using custom name spaces if possible – they cannot be removed from the JCR once created.Avoid creating custom node types for the purpose of simplifying JCR queries – our performance tests show that using attribute / value pairs is usally even faster than a query specifying a particular node type. NGA CQ5 Other Useful Nuggets NGA Developer Workstation Configuration
Java 1.6, latest patch
CRXDE rather than Maven + Eclipse – recommendation from Adobe since small team
CRXDE can be slow and SVN integration is barely sufficient
Missing CRXDE libraries
Disable virus scanner, particularly during install
4 fast cores + 16GB RAM + fast disk (SSD?) so developer can easily run two instances + tools efficiently – some laptops have only 8 which becomes problematic
local Apache + dispatcher would have been helpful

Scripted build process on shared DEV / build server
Uses curl for all operations
Delete and recreate approach taken
SVN export of apps, design, and environment specific sling:osgiConfig nodes
Bundles are built from source to create JARs
Package created of all app templates and components, bundle JARs, and OSGI configs; package deployed to targets NGA CQ5 Other Useful Nuggets For art object imagery, we attempted to:
Use CQ5 DAM to create renditions, including zoom images
Synchronize imagery with the CQ5 DAM using independent Java program invoking JCR APIs over RMI.


Using DAM for rendition generation ballooned the JCR to over 250GB very quickly, particularly during the prototyping stage when we processed the same images multiple times – data store GC would have helped with that, but we didn’t know about it at the time

Insufficient control options over renditions - color profiles are not retained during resizing operations and no out of the box control over resizing algorithms exists; we would have to create our own custom workflow code for image resizing and color profiling operations

rsync over webDAV didn’t work on linux, so we had to prototype a standalone program to interface with the JCR over webDAV (since RMI isn't available in CQ 5.5)

After concluding that the DAM option, while feasible, introduced significant risk and JCR size, we instead:

Enhanced the existing image processing jobs to produce additional sizes; rsync images and serve from a dedicated image and image zoom server
Enhanced our Art Object API provided in the OSGI bundle to include image meta-data
Introduced a dedicated image and image zoom server into our architecture NGA CQ5 Art Data and Image Integration What we did
As the complexity of the art object requirements grew, we studied the feasibility of loading all art data and image meta-data into memory. We optimized the memory footprint of those structures and found that it was feasible. The art data bundle loads and caches in RAM approximately 1GB of data. Searches, faceting, and sorting of this data through a custom API is lightening fast.

What about free text search? We realized that re-implementing the features of Lucene to support free-text search was not something we wanted to do, so we also built a small module to replicate a significantly flattened set of art data in the JCR for Lucene to perform free-text searches of art data. This data set supports both the site search as well as some of the searches on our “advanced collection search” page NGA CQ5 Art Data and Image Integration NGA Art Data – key content for the web site
110k+ art objects, 20k artists, other entities – highly complex relationships stored in external relational database tables
Redesigned web site has complex art data searches, sort orders, and faceting of art data search results
Data must be synchronized – Art object data changes regularly and new objects are added nearly every day

Images of Art Objects – key to a visually rich web site
Images of the art – zoom files are very large, multiple sizes are required, color profile management, specific algorithms required for resizing, must support addition of manually cropped renditions
Image associations with art objects are managed in a different system
Images must be synchronized – images change regularly and new images are added frequently NGA CQ5 Art Data and Image Integration Secure URLs
Need HTTPS for certain form submissions
Avoid abuse of forms that generate e-mail
Lessons Learned
Force HTTPS using Apache and / or with JSPs (if possible) – still figuring out
Use captcha to avoid getting on an e-mail blacklist

User Form Submissions
Need removal of data from publish instances due to gov’t privacy rqmnts
Lessons Learned
Relatively straightforward workflows and workflow launchers can achieve this (auto-reverse-replication and content deactivation); form authors need training
Potential security issue with out of the box form handler (path specification for form element name) – need to lock down permissions for anonymous user and public authenticated users NGA CQ5 Technical Architecture Author Instance Authentication
NGA looked briefly at SSO with Apache / mod_auth* modules - we use it for other web-based systems; the security model (trusted credential) has been deprecated due to potentially serious security problems
NGA opted for LDAP “out of the box” / we will be transitioning to LDAPS now that it’s working over unsecured LDAP.
Lessons Learned
CQ5 trusted credentials attribute is being phased out as indicated in JackRabbit log files – don’t depend on it - unclear what, if any, mechanism will replace it.
That said, Kerberos (or most any other kind of Apache supported authentication) can be integrated with Apache simply by setting a certain HTTP header in the request after Apache authentication is performed. However, there are security issues with this since headers can be spoofed easily – under current SSO, you must not expose CQ5 instances outside of the local machine.
LDAP (prior to CQ 5.6 I believe) has a bug related to case sensitivity of user names – it’s fixed by a patch – OR – users need to kill the session cookies that are created after their initial login using LDAP. NGA CQ5 Technical Architecture Dispatcher Configuration
NGA using ver 4.1.2, but 4.1.3 available for download
Lessons Learned
Dispatcher is an application firewall, so it’s probably best to filter out by default and include only the URL patterns that are needed
Check for the text “reject” in dispatcher.log to identify requests that were rejected by the dispatcher filters
Differentiating between the scheme (http vs. https) of the original browser request is impossible to do when proxying the connection to CQ (either with mod_proxy or dispatcher) over http only because the scheme reported by JSP is the scheme of the proxy, not the browser.
Latest dispatcher (4.1.3?) claims to support native HTTPS without require stunnel. Even so, the latest dispatcher still might not support multiple configurations within a single Apache instance, so you might have to install two separate Apache instances on the same machine to detect the scheme properly.
The dispatcher options for flushing the cache are quite limited – you might have to write your own CGI to intercept dispatcher flush requests and handle them yourself – or go with a very small number of stat file levels – we’re still figuring this out. NGA CQ5 Technical Architecture CQ5 Clustering
Set up is easy, but result is risky (CQ5 admin class) and expensive
When clustering, understand the types of clustering available and make the right choice based on your requirements (e.g. shared JCR files or synchronized copies).
Frequently for small to medium operations teams, the complexities inherent in software clusters actually create more downtime than those clusters save due to other failures.
Active / active requires two CQ5 licenses; active / passive might not (ask Adobe). NGA CQ5 Technical Architecture Installation / Upgrade Lessons Learned
CQ 5.4 to 5.5 is a major change – everything is OSGi – some configs buried and some features no longer supported, e.g. RMI no longer seems like an option.
Java 1.7 not supported by CQ 5.5 – unsure about 5.6 – Adobe claims Oracle has a problem in 1.7 that needs fixing
Always disable virus scanner prior to CQ5 install, particularly McAfee, as CQ5 initial install will not complete if McAfee on-access protection is enabled. Maybe run DEV, QA, and Prod on Linux to avoid having to install a virus scanner ?
webDAV doesn’t seem to be completely supported; compounded by Windows 7 shipping with broken webDAV client.
Lucene index occasionally misses nodes – limited to DEV / QA systems? Reindexing may be an option – we have succesfully reindexed TAR PM, but have not attempted Lucene reindex - heard it takes a while.
Start / stop scripts need modification to work properly on Linux
5.6 seems to have some user interface screens missing, e.g. under Tools. We’re sticking with 5.5 SP 2.1 for now – any opinions about 5.6?
After fresh install, one finds quickly that default memory is insufficient NGA CQ5 Technical Architecture NGA Public Web Site Redesign
CQ5 Lessons Learned
March 27, 2012

David Beaudet
NGA Lead Developer / Architect)
d-beaudet (a.t.) nga.gov In hindsight and assuming availability of resources and time, I would have:
Specified SOLR as our search solution from the start
Sent a few people to CQ5 advanced developer training earlier in the process
Sent a few people to CQ5 admin training earlier in the process
Committed to Maven + Eclipse rather than using CRXDE for development
Would look more seriously at custom oAuth effort rather than using CQ5’s social collaboration module for authentication NGA CQ5 Other Useful Nuggets Lessons Learned
Let CQ do what CQ is good at – authoring, storing, and rendering web content. Resist the temptation to feed a lot of other data sources into the JCR.
Spend the money to consult with an Adobe Architect – it’s worth it. Just be sure they understand how many hours you’ve allocated to each topic so you don’t blow the budget.
Consider using a search engine such as SOLR to power your search features if you have requirements bordering on the complex or if you find yourself struggling with CQ5 JCR queries to get what you need.
The CQ5 DAM is useful for assisting content authors, but until it evolves into something a bit more mature, limit it’s use to the web content authoring process.
Perform data store garbage collection on a regular basis if you find your JCR is growing too rapidly. Remember that CQ backups double your disk space requirement.
Adobe still has a lot of work to do with respect to JCR search. Hopefully the next major version of CQ5 will expose a much richer and better performing search engine to developers, possibly even bundling an integrated SOLR for high-performing facets and searches. This is a gaping hole in the solution and many web sites are gravitating towards faceted browse as a standard feature and a feature that handles a high proportion of user requests. NGA CQ5 Art Data and Image Integration For art object data, we attempted to build a prototype that would feature:
automatic synchronization of data between relational database and JCR
a JCR content hierarchy able to accommodate 1 million+ art data entities and relationships between them without sacrificing data richness or overloading the JCR
Efficient JCR SQL2 queries that return distinct result sets in a highly customized sort order along with custom facets


Relational database to JCR mapping isn’t straightforward. We realized that in order for JCR queries to perform well and in order to avoid duplicate search results, we would have to store multiple copies of the art object data in multiple JCR hierarchies, possibly many depending on the specific queries. The NGA determined that any solution involving duplication of data into an unknown number of content hierarchies was impractical and not extensible.

Custom facet extractors in CQ5 must use the QueryBuilder API – none of the other search APIs support it – and the QueryBuilder implementation requires that all nodes be visited to calculate facets, suggesting that facets are not pre-calculated for nodes - a feature that would seem obvious for any high-performance faceted search engine.

The NGA and other customers have noted that JCR query performance degrades proportionally to the result set size - this suggests that the JCR permissions are checked for every node in the result set - another likely performance killer - permissions should either be cached as well or checking them should be optional for cases when high performance is required.

NGA CQ5 Art Data and Image Integration Publish Instance Authentication
We support native CQ5 users and social users for public authentication
Native CQ5 users are the same users used by the WCM tools, so be sure the default group assignments do not include WCM permissions
oAuth uses the deprecated trusted credentials attribute – so error log fills with warnings about deprecation of trusted credentials attribute
Lessons Learned
oAuth is not a trivial endeavor – there are lots of details and configurations to keep up with plus you’ll be writing some custom code to secure it anyway.
Create specific logging configuration for the oAuth module to send repetitive deprecation warnings to a separate log file or to /dev/null
CQ5.5 SP 2.1 breaks oAuth due to ACL on /etc/cloudservices – you have to adjust it to permit read to /etc/cloudservices/facebook and twitter directories.
HTTPS vs. HTTP and cookie security – might have to get Apache involved to logout users if unsecured cookie transmission is detected – we’re still working on this. NGA CQ5 Technical Architecture Web Server Configurations
NGA uses CQ5 behind Apache – HTTPS for author, both schemes for publish – dispatcher only used for publish.
Lessons Learned
Configure mod_proxy properly to prevent Apache from becoming an open proxy server
SSL configuration in CQ5 requires installing a server certificate into the Java keystore. Unfortunately, Oracle doesn’t include the full set of CA certs that browers do, so your CA’s cert might have to be added to the Java configuration. Moreover, Java updates can replace the keystore which means you have to reinstall the CA cert every time Java is updated. When getting your server’s CSR signed, select a vendor whose CA cert is included out of the box in Java.
Consider creating Apache and dispatcher configuration file templates and automating the creation and distribution of the configs to ensure configuration consistency among environments.
Use Apache rather than CQ5 for index page redirection; generally speaking, minimize the number of requests that have to be sent to CQ5 publish instance
Urge developers to test through DEV dispatcher before requesting code promotion as dispatcher filter rejects are easy to identify; create a CGI to expose a long tail of DEV author, DEV publish, and DEV dispatcher logs so developers can see errors. NGA CQ5 Technical Architecture CQ5 Server Configuration and Operational Concerns
one RHEL 6 VM guest for each of DEV and QA (16 GB RAM / 4 CPUs)
one RHEL 6 VM guest each for prod author and prod publish (16 GB / 4 CPUs)
Lessons Learned
allocate plenty of memory to the JVM for each instance (we use between 4 and 8 GB depending) – use 64 bit operating system otherwise processes are limited to 4GB each
allocate plenty of storage up front – CQ doesn’t perform JCR garbage collection by default – CQ5 halts itself when insufficient storage space is detected – 100GB min
change the default temp directory of CQ5 in start script configuration to a volume with plenty of space
tar persistence is highly I/O intensive - use fastest reliable storage – anyone using SSD? We are using iSCSI on 1 Gbps network
Java patches need to be managed – system library links are severed as part of RHN updates which can lead to CQ5 errors if CQ5 is left running during the patch.
Schedule datastore garbage collection and use incremental backups to minimize impact of backups NGA CQ5 Technical Architecture Technical Architecture
Art Data and Image Integration
Other Useful Nuggets Topics Multiple run modes: Configure your CQ5 instances with multiple run modes (e.g. dev,author) and use sling:osgiConfig nodes to target specific run-mode specific directories (e.g. config.dev.author, config.dev, config.author)
Avoid Felix console: don’t apply configurations outside of your DEV activities using the Felix console – it’s better to define those in osgi configs and push them as part of your build. Also, don’t mix modes of applying these configurations – it gets confusing otherwise as changes are persisted differently by each.
Avoid memory caching of JCR content if possible (for performance reasons, it’s not always possible) and focus instead on the dispatcher to cache your slowest running pages
Use AJAX to avoid cache flushing: For content shared across the site via base page templates (e.g. site-wide alerts), consider using a separate AJAX call in order to avoid having to flush all of your pages from cache. Instead, just cache the individual component instance.
Use reference components and train content authors how to create and use reference component instances to avoid having to duplicate a lot of identical content across your site. If you’re programmatically including fixed path reference components in a template, consider keeping that content isolated to a specific directory that requires a developer to change that content; e.g. we use reference components outside of the context of a cq:page for drawers on our event pages. NGA CQ5 Other Useful Nuggets
Full transcript