Linear Review is an Outdated Methodology

As we trudge through the first week of 2011, I am going through my list of Blog posts that I wanted to comment on and the December 28, 2010 post on Linear Review by Venkat Rangan, Clearwell Systems CTO, seemed like a good place to start 2011.  The post titled, “Reinventing Review in Electronic Discovery”  discusses a topic that I am very familiar with and have been somewhat outspoken about it in the past couple of years.  Review costs still comprise over 70% of the overall cost of eDiscovery and therefore as an industry, we need to find better ways to approach review and, more importantly, reduce the costs.

Given my background in enterprise class applications development methodology and technology, I lived through the paradigm shift when that industry shifted from legacy waterfall methodology (i.e. linear) to rapid applications development (RAD) and now agile development methodology.  The increases in productivity were dramatic.

Mr. Rangan’s bases his Blog post on a excellent paper by The Demise of Linear Review by Bennett Borden of Williams Mullen.  Mr. Rangan states that the paper, citing several factual data from various studies, as well as drawing parallel to other similar anachronisms of the past, makes excellent arguments for rethinking how legal review is performed in eDiscovery.

I hope that in 2011, the litigation market begins to understand and embrace both the practical and financial benefits of replacing linear review with newer and more effective review methodologies and technologies.

The full text of Mr. Rangan’s Blog post is as follows:

In a recent workshop that I attended, I had the privilege of sharing thoughts on the latest electronic discovery trends with other experts in the market. Especially interesting to me was discussing the provocatively titled paper, The Demise of Linear Review by Bennett Borden of Williams Mullen. The paper, citing several factual data from various studies, as well as drawing parallel to other similar anachronisms of the past, makes excellent arguments for rethinking how legal review is performed in e-discovery.

When linear review is mentioned, the first mental picture one conjures up is boredom. It has generally been associated with a mental state that is a result of repetitive and monotonous tasks, with very little variation. To get a sense for how bad this can affect performance, one only needs to draw upon several studies of boredom at the workplace, especially in jobs such as mechanical assembly of the 1920s and the telephone switchboard operators of the 1950s. In fact, the Pentagon sponsored study, Implications for the design of jobs with variable requirements, from Navy Personnel Research and Development Center, presents an excellent treatise on contributors for workplace fatigue, stress, monotony, and distorted perception of time. This is best illustrated in their paper:

Mechanical assembly, inspection and monitoring, and continuous manual control are the principal kinds of tasks most frequently studied by researchers investigating the relationship between performance and presumed boredom. On the most repetitive tasks, degradation of performance has typically been found within 30 minutes (Fox & Embry, 1975; Saito, Kishida, Endo, & Saito, 1972). The early studies of the British Industrial Fatigue Board (Wyatt & Fraser, 1929) concluded that the worker’s experience of boredom could be identified by a characteristic output curve on mechanical assembly jobs. The magnitude of boredom was inversely related to output and was usually marked by a sharp decrement in the middle of a work period.
How does this apply to linear review? Well, a linear review is most often performed using a review application or tool, simulating a person reading and classifying a pile of documents. The reviewer is asked to read the document and apply a review code, based on their judgment. While it appears easy, it can be one of the most stressful, boring, and thankless jobs for a well-educated, well-trained knowledge worker. Even with technology and software advances a reviewer is required to read documents in relatively constrained workflows. Just scrolling through pages and pages of a document, comprehending its meaning and intent in the context of the production request can make it stressful. To add to this, reviewers are often measured for their productivity based on the number of documents or pages they review per day or per hour. In cases where large number of reviewers are involved, there are very direct comparisons of rates of review. Finally, the review effort is judged for quality without consideration for the very elements that impact quality. Imagine a workplace task where every action taken by a knowledge worker is monitored and evaluated to the minutest detail.

Given this, it is no wonder that study after study has found a straight plough-through linear review produces less than desirable results. A useful way to measure effectiveness of a review exercise is to submit the same collection of documents to multiple reviewers and assess their level of agreement on their classification of the reviewed documents in specific categories. One such study, Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, finds that the level of agreement among human reviewers was only in the 70% range, even when agreement is limited to positive determination. As noted in the study, previous TREC inter-assessor agreement notes as well as other studies on this subject by Barnett et al., 2009 also shows a similar and consistent result. Especially noteworthy from TREC is the fact that only 9 out of 40 topics studied had an agreement level higher than 70%, while remarkably, four topics had no agreement at all. Some of the disagreement is due to the fact that most documents fall on varying levels of responsiveness which cannot easily be judged on binary yes/no decision (i.e., the “where do you draw the relevance line” problem). However, a significant source on variability is simply attributed to the boredom and fatigue that comes with repetitiveness of the task.

A further observation on reviewer effectiveness is available from the TREC 2009 Overview Report, which studied the appeals and adjudication process of that year’s Interactive Task. This study offers an excellent opportunity to assess the effectiveness of initial review and subsequent appeals and adjudication process. As noted in the study, the Interactive Task involves an initial run submission from participating teams which are sampled and reviewed by human assessors. Upon receiving their initial assessments, participating teams are allowed to appeal those judgments. Given the teams’ incentive to improve upon the initial results, they are motivated to construct an appeal for as many documents as they can, with each appeal containing a justification for re-classification. As noted in the study, the success rates of appeals were very high, with 84% to 97% of initial assessments being reversed. Such reversals were across the board and directly proportional to the number of appeals, suggesting that even the assessments that were not appealed could be suspect. Another aspect that is evidenced is that the appeals process requires a convincing justification from the appealing team, in the form of a snippet of the document, document summary, or a portion of the document highlighted for adjudication. This in itself biases the review and makes it easier for the topic assessor to get a clearer sense for the document on their attempt at adjudicating the appeal. This fact is also borne out by the aforementioned Computer Classification vs. Manual Review study where the senior litigator with the knowledge of the matter had the ability to offer the best adjudications.

Given that linear review is flawed, what are the remedies? As noted in Bennett’s paper, intelligent use of newer technologies along with a review workflow that leverages them can offer gains that are demonstrated in other industries. Let’s examine a few of them.

Response Variation Response variation is a strategy for coping with boredom by attempting to build variety into the task itself. In mechanical assembly lines, response variation is added through innovative floor and task layouts, such as Cellular Layout. On some tasks, response variation may involve only simple alternation behaviors, such as reversing the order in which subtasks are performed; on others, the variety may take more subtle forms reflected in an inconsistency of response times. In the context of linear review, it can help to organize your review batches so that your review teams alternate classifying documents for responsiveness, privilege and confidential etc. Another interesting approach would be to mix the review documents but suggest that each be reviewed for a specific target classification.

Free-Form Exploration Combining aspects of early case assessments and linear review is one form of exploration that is known to offer both a satisfying experience and effective results. While performing linear review, the ability to suspend the document being reviewed and jump to other similar documents and topics gives the reviewer a cognitive stimulus that improves knowledge acquisition. Doing so offers an opportunity for the reviewer to learn facts of the case that would normally be difficult to obtain, and approach the knowledge levels of a senior litigator of the case. After all, we depend on the knowledge of the matter to be a guide for reviewers, so attempts to increase their knowledge of the case can only be helpful. Also, on a free-form exploration, a reviewer may stumble on an otherwise difficult to obtain case fact and the sheer joy of finding something valuable would be rewarding.

Expanding the Work Product Besides simply judging the review disposition of a document, the generation of higher value output such as document summaries, critical snippets, and document meta-data that contribute to the assessment can both reduce the boredom of the current reviewer as well as contribute valuable insights to other reviewers. As noted earlier, being able to assist the review with such aids can be immensely helpful in your review process.

Review Technologies Of course, fundamentally changing linear review with specific technologies that radically changes the review workflow is an approach worth considering. While offering such aids, it must be remembered that human judgment is still needed and the process must incorporate both increasing their knowledge as well as their ability to apply judgment. We will examine these technologies in an upcoming post.

Labels: 26(f), Discovery, Early Case Assessment, ECA, EDD, eDiscovery, eDiscovery Best Practices, eDiscovery Costs, eDiscovery Market, EDRM, ESI

posted by Charles Skamser @ 10:08 AM 0 Comments

Is the Enterprise Market Eluding the Legacy Litigation Technology Providers?

Gartner foresees that worldwide eDiscovery software revenues will reach $1.2 billion in 2010, an increase of 23% over 2009. Further, according to Gartner, U.S. companies alone will spend $29.8B on GRC activities in 2010, up 3.9% from 2009.  And, just about every analyst in the world is predicting that a large percentage of these revenues will come from the enterprise.   However, I haven’t seen the legacy litigation technology providers taking advantage of this opportunity.  I believe that there are several reasons.

First of all, it wasn’t that long ago that many of these organizations were still trying to  figure out how to move from selling paper processing services to eDiscovery services and the associated technology.  It is possible that it just too soon or too much to ask them to make another paradigm shift to selling into the enterprise.

The second and biggest reason is that these legacy providers do not have sales and marketing organizations that understand how to sell into the legal and IT departments of the corporation.  And, there is a big difference between the sales cycle of selling to paralegals and litigation service groups within law firms and the complex sales cycle of an enterprise.  Where Friday donuts and a few happy hours did the trick for the law firms, the enterprise requires value propositions, ROI analysis, competitive analysis and an understanding of how the procurement process works.

The third reason is the lack of understanding of what actually goes on within the enterprise IT department and why they buy stuff.  Law firms are very much case based and/or expense based buyers as they want to pass on all cost to their clients.   Enterprises, on the other hand, are more into capital expenditures, total cost of ownership and amortizing costs over multiple cost centers for long periods of time.   Law firms have partners that can make quick buying decisions (especially if they have a big case and/or a big insurance company that is paying the bills).  Enterprises, on the other hand,  make decisions through multiple stakeholders that sit on multi-divisional committees that have to negotiate budget commitments and coordinate budget cycles.

Finally, at least for the purposes of this Blog post, law firms for the most part are comfortable with independent technology platforms with little or no integration.   In fact, there is an argument to be made that some law firms actually like technology solutions that require manual intervention that enables increased billable hours.  Enterprises, on the other hand are much more interested in integrated enterprise wide solutions that automate entire business processes.

In the end, the legacy litigation technology providers that survive will probably be absorbed into technology vendors that are providers a much broader base of Governance, Risk and Compliance (GRC) processing and analytics.  In the meantime, the current players in the litigation technology market better figure out how to sell to the buyers that are actually buying technology.

Labels: eDiscovery, eDiscovery Market, eDiscovery Software Sales, GRC, Total Cost of Ownership

posted by Charles Skamser @ 9:12 PM 1 Comments

When to Bring eDiscovery In-house or Move to Cloud Computing?

Over the past 18 months, the “big buzz” in the eDiscovery market has been that the Global 2000 are bringing eDiscovery in house.  The value proposition is that outsourcing to legal vendors, service providers and/or outside counsel is just too expensive.  And, ultimately, ensuring that it (eDiscovery) is done properly (i.e. in a legally defensible manner, etc.) is the responsibility of the General Counsel (GC) and other C level executives (i.e. CIO, CFO, CEO) anyway.

However, what about members of the Global 2000 that don’t have enough litigation to justify bringing it (eDiscovery) in-house?  And, what about all the rest of the enterprises outside of the Global 2000 (the majority of the enterprises worldwide) that never have to deal with the issues and costs of eDiscovery?

First of all, with the inevitable convergence of eDiscovery and Governance, Risk and Compliance (See “ Government Intervention and Oversight Driving the Convergence of eDiscovery and Governance, Risk and Compliance (GRC)”) all enterprises worldwide are going to have to deal with the issues of information management and reporting as it relates to eDiscovery and GRC.
So, just because you never have to respond to requests to produce information due to litigation, it is negligent to not  be prepared (e.g. have a data retention policy and a plan for eDiscovery).  And now, with the accelerating increase in government intervention and oversight, Governance, Risk and Compliance (GRC) are almost certainly going to affect all enterprises worldwide in some way shape or form.

So, given the cost of “outsourcing” along with this inevitable new playing field, when do you bring eDiscovery in-house? Or, given the convergence of eDiscovery and GRC, maybe a better question is when should the enterprise have an  in-house plan and associated capabilities to support whatever legal and/or GRC requests come along?

The answer for some may be to keep outsourcing.  However, those that do had better brush up on their responsibilities (outsourcing or not) as the courts are no longer accepting ignorance as a defense.

Another alternative maybe to investigate Cloud Computing as a cost effective way of moving eDiscovery / GRC in-house without having to invest in all of the IT infrastructure.  Obviously, archiving data (e.g. email) “in the Cloud” has matured to the point where it is almost a “no brainer” for most enterprises to consider (no hate mail from the anti Cloud contingency please as your security concerns are beginning to wear a bit thin).  And, most if not all of the applications that it takes to support eDiscovery / GRC processing such as Early Case Assessment (ECA), Legal Holds, Search and Analytics and Document Review are now available as Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS).  Moving large amounts of data around and maintaining an appropriate chain-of-custody are still issues that need to be watched closely in this environment.

Enterprises worldwide, whether in or outside the Global 2000, are having to deal with the paradigm shift and subsequent realities of eDiscovery and Governance, Risk and Compliance (GRC) reporting.   And, the pressures of the worldwide financial crisis along with the accelerating increase in government intervention and oversight have unfortunately added another layer of complexity.

However, Cloud Computing is now a very viable alternative when considering moving your eDiscovery and GRC operations in-house.

Labels: Cloud Computing, Compliance, Convergence, Early Case Assessment, eDiscovery, eDiscovery Costs, eDiscovery Market, Electronically Stored Information, Governance, GRC, Risk and Compliance

posted by Charles Skamser @ 9:16 AM 0 Comments

2008 Financial Crisis Not Slowing Investment in Litigation Technology Market

Having helped raise in excess of $75 million in my career for a variety of high tech software startups I was not surprised to learn that Recommind, which offers an “eDiscovery” service aimed at law firms, has raised $7.5 million in a first round of venture funding. Although the funding was announced on October 8, 2008 and has probably been in the works for some time, there is always money available for a solid technology investment. Further and as indicated on this Blog over the past couple of weeks, the Financial Crisis of 2008 will actually provide a windfall for technology vendors in the eCompliance and eDiscovery markets.
The full text of the Recommind announcement as seen on the the Venture Beat site is as follows:

Recommind, which offers an “eDiscovery” service aimed at law firms, has raised $7.5 million in a first round of venture funding. Despite the slowing economy, chief executive Robert Tennant says his company should stay very busy in the coming months.
That’s because San Francisco-based Recommind provides a key tool for litigation. Its Axcelerate product is used during electronic discovery, also known as eDiscovery, which is the initial process of sorting through electronic records. With financial institutions collapsing, and legal battles likely to follow, the need for eDiscovery may be on the rise. Tennant says Recommind has been getting increased requests from its law firm customers, and he expects more to come, even as many companies cut back on their costs.
“Some technology companies’ capital decisions will get delayed … but when a judge says, ‘Thou shalt produce,’ you have to produce,” he says.
Of course, there are other startups entering the market — either involved directly in eDiscovery or in archiving documents for legal purposes — and they’re getting funding, too. For example, Mimosa Systems raised $17 million earlier this year. Other competitors include eDiscovery company Attenex and enterprise search company Autonomy. (Recommind also provides enterprise search.)
But Tennant says Recommind’s service is particularly advanced. He calls it “eDiscovery 2.0″ — a phrase Tennant sounds slightly embarrassed to use, but which gets the idea across. Recommind’s Axcelerate automates the initial process of sorting through documents and judging their relevance, thus making eDiscovery faster and more affordable.
Recommind has also been in the business for a while, with customers like well-known legal firms Cooley Godward Kronish and Wilson Sonsini Goodrich and Rosati. It’s already profitable, and has been growing at about 100 percent per year, Tennant says. So why does it need the funding? There’s a big opportunity for growth, he says. Recommind will use the money to expand its sales team and also acquire some new technology.
The round was let by Kennet Partners.

Labels: 2008 Financial Crisis, Autonomy, eDiscovery Market, eDiscovery Windfall, Mimosa Systems, Recommind, Venture Capital

posted by Charles Skamser @ 4:21 PM 0 Comments