Linear Review is an Outdated Methodology

As we trudge through the first week of 2011, I am going through my list of Blog posts that I wanted to comment on and the December 28, 2010 post on Linear Review by Venkat Rangan, Clearwell Systems CTO, seemed like a good place to start 2011.  The post titled, “Reinventing Review in Electronic Discovery”  discusses a topic that I am very familiar with and have been somewhat outspoken about it in the past couple of years.  Review costs still comprise over 70% of the overall cost of eDiscovery and therefore as an industry, we need to find better ways to approach review and, more importantly, reduce the costs.

Given my background in enterprise class applications development methodology and technology, I lived through the paradigm shift when that industry shifted from legacy waterfall methodology (i.e. linear) to rapid applications development (RAD) and now agile development methodology.  The increases in productivity were dramatic.

Mr. Rangan’s bases his Blog post on a excellent paper by The Demise of Linear Review by Bennett Borden of Williams Mullen.  Mr. Rangan states that the paper, citing several factual data from various studies, as well as drawing parallel to other similar anachronisms of the past, makes excellent arguments for rethinking how legal review is performed in eDiscovery.

I hope that in 2011, the litigation market begins to understand and embrace both the practical and financial benefits of replacing linear review with newer and more effective review methodologies and technologies.

The full text of Mr. Rangan’s Blog post is as follows:

In a recent workshop that I attended, I had the privilege of sharing thoughts on the latest electronic discovery trends with other experts in the market. Especially interesting to me was discussing the provocatively titled paper, The Demise of Linear Review by Bennett Borden of Williams Mullen. The paper, citing several factual data from various studies, as well as drawing parallel to other similar anachronisms of the past, makes excellent arguments for rethinking how legal review is performed in e-discovery.

When linear review is mentioned, the first mental picture one conjures up is boredom. It has generally been associated with a mental state that is a result of repetitive and monotonous tasks, with very little variation. To get a sense for how bad this can affect performance, one only needs to draw upon several studies of boredom at the workplace, especially in jobs such as mechanical assembly of the 1920s and the telephone switchboard operators of the 1950s. In fact, the Pentagon sponsored study, Implications for the design of jobs with variable requirements, from Navy Personnel Research and Development Center, presents an excellent treatise on contributors for workplace fatigue, stress, monotony, and distorted perception of time. This is best illustrated in their paper:

Mechanical assembly, inspection and monitoring, and continuous manual control are the principal kinds of tasks most frequently studied by researchers investigating the relationship between performance and presumed boredom. On the most repetitive tasks, degradation of performance has typically been found within 30 minutes (Fox & Embry, 1975; Saito, Kishida, Endo, & Saito, 1972). The early studies of the British Industrial Fatigue Board (Wyatt & Fraser, 1929) concluded that the worker’s experience of boredom could be identified by a characteristic output curve on mechanical assembly jobs. The magnitude of boredom was inversely related to output and was usually marked by a sharp decrement in the middle of a work period.
How does this apply to linear review? Well, a linear review is most often performed using a review application or tool, simulating a person reading and classifying a pile of documents. The reviewer is asked to read the document and apply a review code, based on their judgment. While it appears easy, it can be one of the most stressful, boring, and thankless jobs for a well-educated, well-trained knowledge worker. Even with technology and software advances a reviewer is required to read documents in relatively constrained workflows. Just scrolling through pages and pages of a document, comprehending its meaning and intent in the context of the production request can make it stressful. To add to this, reviewers are often measured for their productivity based on the number of documents or pages they review per day or per hour. In cases where large number of reviewers are involved, there are very direct comparisons of rates of review. Finally, the review effort is judged for quality without consideration for the very elements that impact quality. Imagine a workplace task where every action taken by a knowledge worker is monitored and evaluated to the minutest detail.

Given this, it is no wonder that study after study has found a straight plough-through linear review produces less than desirable results. A useful way to measure effectiveness of a review exercise is to submit the same collection of documents to multiple reviewers and assess their level of agreement on their classification of the reviewed documents in specific categories. One such study, Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, finds that the level of agreement among human reviewers was only in the 70% range, even when agreement is limited to positive determination. As noted in the study, previous TREC inter-assessor agreement notes as well as other studies on this subject by Barnett et al., 2009 also shows a similar and consistent result. Especially noteworthy from TREC is the fact that only 9 out of 40 topics studied had an agreement level higher than 70%, while remarkably, four topics had no agreement at all. Some of the disagreement is due to the fact that most documents fall on varying levels of responsiveness which cannot easily be judged on binary yes/no decision (i.e., the “where do you draw the relevance line” problem). However, a significant source on variability is simply attributed to the boredom and fatigue that comes with repetitiveness of the task.

A further observation on reviewer effectiveness is available from the TREC 2009 Overview Report, which studied the appeals and adjudication process of that year’s Interactive Task. This study offers an excellent opportunity to assess the effectiveness of initial review and subsequent appeals and adjudication process. As noted in the study, the Interactive Task involves an initial run submission from participating teams which are sampled and reviewed by human assessors. Upon receiving their initial assessments, participating teams are allowed to appeal those judgments. Given the teams’ incentive to improve upon the initial results, they are motivated to construct an appeal for as many documents as they can, with each appeal containing a justification for re-classification. As noted in the study, the success rates of appeals were very high, with 84% to 97% of initial assessments being reversed. Such reversals were across the board and directly proportional to the number of appeals, suggesting that even the assessments that were not appealed could be suspect. Another aspect that is evidenced is that the appeals process requires a convincing justification from the appealing team, in the form of a snippet of the document, document summary, or a portion of the document highlighted for adjudication. This in itself biases the review and makes it easier for the topic assessor to get a clearer sense for the document on their attempt at adjudicating the appeal. This fact is also borne out by the aforementioned Computer Classification vs. Manual Review study where the senior litigator with the knowledge of the matter had the ability to offer the best adjudications.

Given that linear review is flawed, what are the remedies? As noted in Bennett’s paper, intelligent use of newer technologies along with a review workflow that leverages them can offer gains that are demonstrated in other industries. Let’s examine a few of them.

Response Variation Response variation is a strategy for coping with boredom by attempting to build variety into the task itself. In mechanical assembly lines, response variation is added through innovative floor and task layouts, such as Cellular Layout. On some tasks, response variation may involve only simple alternation behaviors, such as reversing the order in which subtasks are performed; on others, the variety may take more subtle forms reflected in an inconsistency of response times. In the context of linear review, it can help to organize your review batches so that your review teams alternate classifying documents for responsiveness, privilege and confidential etc. Another interesting approach would be to mix the review documents but suggest that each be reviewed for a specific target classification.

Free-Form Exploration Combining aspects of early case assessments and linear review is one form of exploration that is known to offer both a satisfying experience and effective results. While performing linear review, the ability to suspend the document being reviewed and jump to other similar documents and topics gives the reviewer a cognitive stimulus that improves knowledge acquisition. Doing so offers an opportunity for the reviewer to learn facts of the case that would normally be difficult to obtain, and approach the knowledge levels of a senior litigator of the case. After all, we depend on the knowledge of the matter to be a guide for reviewers, so attempts to increase their knowledge of the case can only be helpful. Also, on a free-form exploration, a reviewer may stumble on an otherwise difficult to obtain case fact and the sheer joy of finding something valuable would be rewarding.

Expanding the Work Product Besides simply judging the review disposition of a document, the generation of higher value output such as document summaries, critical snippets, and document meta-data that contribute to the assessment can both reduce the boredom of the current reviewer as well as contribute valuable insights to other reviewers. As noted earlier, being able to assist the review with such aids can be immensely helpful in your review process.

Review Technologies Of course, fundamentally changing linear review with specific technologies that radically changes the review workflow is an approach worth considering. While offering such aids, it must be remembered that human judgment is still needed and the process must incorporate both increasing their knowledge as well as their ability to apply judgment. We will examine these technologies in an upcoming post.

Labels: 26(f), Discovery, Early Case Assessment, ECA, EDD, eDiscovery, eDiscovery Best Practices, eDiscovery Costs, eDiscovery Market, EDRM, ESI

posted by Charles Skamser @ 10:08 AM 0 Comments

Records Management and Early Case Assessment (ECA)

• Identify the specific business objectives for their records management initiative and select the right vendor, as a records management program requires a long-term commitment and the product needs to be compatible with the organization's content management strategy.
• Assess the records management product's capabilities to integrate and extend records retention policy into other content repositories, e-mail systems, document archives and file stores, as well as its search and legal discovery capabilities.
• Consider records management products with capabilities to support multiple content types and media — physical records, which may be in distributed record centers; electronic documents, which may reside in multiple repositories; and e-mail, which may exist in both the e-mail system and e-mail archives.
• Consider the records management product's certification under country- and regional-level standards such as Department of Defense (DoD) Directive 5015.2-STD, Model Requirements for the Management of Electronic Records (MoReq) and Victorian Electronic Records Strategy (VERS)

MarketScope Information management leaders, compliance officers, legal counsel and records managers are working on records management projects that involve managing business document retention and preservation. A good records management initiative includes people, processes and technology. Industry and government regulations have helped lead or even necessitate records management adoption, but e-discovery as well as the need for more effective control of vast information stores in the enterprise have become even more important drivers. While highly regulated industries, such as financial services, utilities and pharmaceuticals, have always had record retention requirements, changes in government regulations have impacted companies across a broader range of industries. Freedom of information legislation, which has proliferated on a global scale, has also increased the need in government organizations to implement records management initiatives, in order to properly retain appropriate documents so they can be produced as requested. Despite today's challenging economic environment, the need for transparency and accountability in government agencies continues to drive the adoption of records management.

Records management systems have evolved from little more than the creation of indexes and tracking mechanisms for the preservation of paper documents to the preservation of electronic documents. As the range of digital content types continues to grow, electronic records management systems should now include mechanisms for dealing with audio, video and Web content. The emergence of wikis, blogs and social networking platforms also requires policies and approaches for retention of user-generated content. E-mail has proved to be especially problematic when it comes to maintaining business records. While e-mail active archiving systems support role-based and event-based retention, records management is still needed for those e-mails that must be retained as records.

Many records management products are integral components of enterprise content management (ECM) suites and will require additional software licenses. The ability to integrate and declare documents from ECM applications should be a prerequisite for records management products. Integration with Microsoft Office is also important, because much of the creation of future business records occurs in this information worker environment. As e-mail and instant messaging are now staples of conducting business, records management products must also be able to integrate with these infrastructure applications. Records management vendors are also emphasizing increased integration with e-mail active archiving, compliance and e-discovery applications. As the diversity of applications and content repositories expands, policy-driven records management, allowing for the automatic classification of records, will become the preferred choice.
Standards have been established in many countries to provide local requirements for records management. These standards include how different record types should be managed, rules for metadata creation, security definitions and implementation guidelines. Many of these standards have released their second versions. Key records management standards include:

• DoD Directive 5015.2-STD — A U.S. standard and certification for records management developed by the Department of Defense. The DoD 5015.2-STD version 2 standard includes a Chapter 2 version and a Chapter 4 version, which detail additional security and classification requirements. The latest version, which records management vendors are now being certified for, is DoD 5015.2-STD version 3.
• MoReq2 — The European Model Requirements for the Management of Electronic Records is a European standard for records management functionality and deployment. The goal of MoReq is to subsume individual national records management standards in the European community. MoReq2, the latest version developed and released in February 2008, is more comprehensive in terms of testing, governance and functionality. Certification and testing of MoReq2 for records management products is just beginning, and as MoReq2 becomes more formalized, we believe that it will serve as a key records management standard.
• TNA2 — TNA2 is a U.K. standard and certification published by The National Archives. Its requirements reflect the legislation and standards for information management and include aspects of ISO 15489, MoReq, e-government and freedom of information. TNA-approved records management products meet compliance with the standards but no certification process is used.
• Document Management and Electronic Archiving (DOMEA) — A German strategy and standard for records management and document management in government agencies. It has seen limited compliance and adoption by records management vendors.
• VERS 2 — The Victorian Electronic Records Strategy is a framework of standards and implementation with the goal of reliably and authentically archiving electronic records created or managed by the Victorian government in Australia. This standard has become the de facto standard for records management projects in Australia.
• Information & Documentation on Records Management Guidelines (ISO 15489) — The first international standard for the implementation of records management. It defines how records should be created, captured and managed, including policies and processes. Many national records management standards include elements of ISO 15489.

These standards can be used as a foundation and guideline for the evaluation and implementation of records management products. The DoD 5015.2 standard is well-established and is the only one that has a formal certification process. It is the de facto benchmark for records management products and acts as certification for essential product functionality. Gartner recommends that enterprises use the DoD 5015.2 STD 2007, version 3 standard as a key requirement when acquiring any records management product, along with consideration of local standards.

Labels: 26(f), Compliance, Early Case Assessment, ECA, eDiscovery, General Counsel, Records Management, Records Rentention

posted by Charles Skamser @ 10:36 AM 1 Comments

Early Case Assessment (ECA)

With the accelerating increase in the amount of Electronically Stored Information (ESI) and the associated changes to the Federal Rules of Civil Procedure (FRCP) enacted in December of 2006, a proper understanding of preservation requirements and Early Case Assessment (ECA) have become an integral part of the litigation process.


In a perfect world, as soon as defense counsel or anyone within an organization with fiduciary responsibilities believes that there is the potential for litigation or is actually notified of a pending litigation, they have the responsibility to preserve any evidence that may pertain to the matter.

While each state, and some local jurisdictions, has promulgated its own rules, the Federal Rules of Civil Procedure (FRCP) in general and FRCP 26(a) and FRCP 26(b) specifically set forth the guidelines for producing, both spontaneously and upon request relevant and material evidence.

Inherent in the reasoning behind these rules is the requirement that in order to produce a thing, one must first have preserved it. For those parties who are not inclined to appreciate this inherent reasoning, there are other rules (and corresponding sanctions) designed to enhance this understanding.

For discussion of the "safe harbor" provisions of a proposed amendment to FRCP 37, and an analysis of numerous federal sanction request cases as related to the advisability of such amendment, see S. Scheindlin and K. Wangkeo, eDiscovery Sanctions in the 21st Century, 11 Mich. TELECOMM. TECH. L. REV. 71 (2004), http://www.mttlr.org/voleleven/scheindlin.pdf

For those parties that do understand the importance of these requirements or have learned the hard way through sanctions or case losses, the importance of these requirements, Early Case Assessment (ECA) is hopefully become a part the standard operating procedure (SOP) as soon as it is clear that a matter is imminent.

This all being said, Early Case Assessment (ECA) is becoming an actual market with its own consultants, technology, services and beginning of best practices.

Industry Quotes and Comments on Early Case Assessment
Following are several quotes from vendors and comments regarding the importance of ECA and their technology. As it appears that just about every technology vendor in the litigation space is getting into ECA in some way, this list is in no way comprehensive and therefore I would encourage other vendors to send me their thoughts, comments and descriptions of their ECA technologies.


Clearwell
-----------
"Early case assessments represent a critical point in the lifecycle of a case, and is one of the primary reasons why leading enterprises and law firms are selecting solutions like Clearwell," said Kamal Shah, vice president of marketing, Clearwell Systems. "Clearwell plays a pivotal role by helping in-house legal teams and law firms assess risk based on case facts, more accurately estimate e-discovery costs, and develop case strategies much earlier in the e-discovery process."

"We were facing a complicated case and had virtually no time to determine the facts. If we did not have access to Clearwell, the case would have become excessively costly and time consuming, resulting in significantly increased risk," said Steven Perfrement, Esq, partner in the Litigation Practice Group at Holme Roberts & Owen LLP. "We were very impressed with the speed and ease of use of the Clearwell platform. The product was very intuitive and user-friendly, which saved us precious time on the case.

Clearwell's advanced capabilities made it very easy to find the key facts needed to accurately evaluate the case and prepare for early settlement negotiations." Clearwell's early case assessment capabilities provide insight into case facts at the beginning stages of an investigation or litigation, and enables enterprises and their law firms to:

- Determine what happened and identify the "key players."
- How did the incident happen?
- Why did it happen?
- Who was involved?
- When did it happen?
- How does the case posture appear on the merits?
- What is the amount of controversial facts and what it the ultimate exposure?
- Is this a routine matter or a unique situation?
- Can the case be settled quickly, or must we prepare for a protracted battle?
- What is the volume of evidence that is pertinent to the case?
- hat are the possible culling and review strategies that can reduce the cost and time of eDiscovery?
- Are you collecting all the data that you are supposed to be collecting?
- Are there any requirements for foreign language expertise?
- Are there any data quality issues?
- Which search term analysis may be necessary to present and discuss during the "meet and confer."?
- Which terms are most important to the case?
- Are the search terms proposed by requesting party too broad?
- Do you have an argument to limit the scope of discovery request?

Daticon
----------
Performing an Early Case Assessment (ECA) can provide valuable insight into all of these areas and better prepare the litigation team for the "meet and confer" with the opposition. A recent industry survey indicated that performing an early case assessment resulted in a favorable outcome in 76% of cases - among other benefits.

A well executed ECA assists the litigator and corporate counsel in understanding the following:

- The custodians involved in the matter
- Where their data is located
- How much data the case involves
- If any of the data is inaccessible

Additionally, in some cases ECA can provide visual insight into the data itself. Accordingly, ECA permits an informed risk assessment of the matter so the case can be settled quickly if potential costs rise or the results of the data visualization reveal bad facts.

Managing Legal Risk Using ECA
I also found a very interesting Blog posting on DCIG by Joshua L. Konkle on March 18, 2008 6:00 am titled Managing legal risk using early case assessment to reduce reviewed documents.

And although I don't completely agree with everything that Gregory Buckles says, it provides yet another voice indicating the importance of ECA. The full text of this posting is as follows:

Synopsis Part 1: Legal and business issues IT must know when choosing an early case assessment tool

Synopsis Part 2: Managing legal risk using early case assessment to reduce reviewed documents Electronic data discovery interview - Gregory "Greg" Buckles, eDiscovery business process consultant, Reason-ed, LLC, (Part 2 of 2).

Gregory Buckles is an independent corporate consultant specializing in discovery technology and process solutions. He has 19 years experience in discovery and litigation including police forensics, law firm, vendor, corporate and software development. He is an active participant in the Sedona Conference and EDRM projects.By Joshua Konkle writing for DCIGInc.comhttp://www.dciginc.com/

Joshua Konkle: I've discussed in-house early case assessment (ECA) with a few vendors. A common response is risk associated in the review process. Risk ensues when during the review an attorney needs more data from the source data set for review purposes, thus requiring the company to revisit the ECA system for more data. I was told in most cases it happens 70% of the time. What are you thoughts on that review data risk?

Greg Buckles: I would agree with that 70% of the time reviews require more data from the source, in fact, it is probably higher. The reason the source data needs to be recalled during review is based on a simple fact - "the review phase is the FIRST time a qualified reviewer has looked at the data qualitatively, i.e. custodians, concepts, context etc."Waiting until the data is in a review system to evaluate it is causing companies thousands if not millions annually. Those dollars would be much better spent as pennies, which is the cost of ECA tools in terms of review budgets. The goal's are simple 1) reduce data sets going into review 2) improve data review during collection.Companies can reduce the amount of data being re-requested during review if they evaluated data ahead of review. If a company desires to reduce the amount of money spent on review they must implement on-premise ECA tools and use them to review data.

Joshua Konkle: In your experience, what is happening today? Aren't companies doing early cases assessment through interviews etc?

Greg Buckles: Companies are doing early case assessment. Often times it is a very simple approach. For example, an attorney working for the company will ask an employee for their opinion on the issue and to submit all documents and email related to the cases or issue. Individual perception and memory impact these responses. Then, when an end-user delivers data it is typically only what they have received and what they have filed. However, there are thousands or more emails and documents in their sent items. These little things are unintentionally overlooked by employees.To overcome these challenges companies need tools that analyze the known relevant data. They don't need complicated preservation and legal hold; they just need to start looking at the data earlier in the process. For example, a recent client used the Axis Deduplicator to deduplicate PST files. Since it the reduced the size, they reduced the cost of an ECA tool. In this example, the client used Attenex Snapshot reports to get a dashboard view of the custodians and concepts. The intuitive interfaces enable a focused view to find the critical facts and criteria needed to decide strategy and arm the client for the meet-and-confer.

Joshua Konkle: In your experience, what are the pitfalls that enterprises have yet to encounter bringing eDiscovery inside their firewall?

Greg Buckles: The primary issue is defining which parts of the process to in-source and the thresholds for using outside assistance. The costs and effort associated with an average discovery must be evaluated. Then a company can design a legal risk management system in line with their needs and expected budgets. A gap analysis for both cost and effort will ensure the economics of the system are in the company's best interests. By economics, I mean avoiding spending more money than they are saving. A good start would be to purchase or test ECA tools.

If you would like to communicate with him directly, he can be reached at greg(at)reason-ed.com or by calling Reason-eD, LLC at 1 713 530 3416.

Labels: 26(f), Early Case Assessment, eCapture, eDiscovery, FRCP Rule 26(f), Meet and Confer, Pretrial Conference

posted by Charles Skamser @ 10:28 AM 1 Comments

Search Terms and FRCP 26(b) (2)(B)

City of Seattle v. Prof'l Basketball Club, LLC, 2008 WL 539809 (W.D. Wash. Feb. 25, 2008) is a great example for all of us to understand the seriousness and importance of detailed preperation for the Rule 26(f) meetings and how the courts (or at least this court) views the agreements reached between the parties at this meetings.

Overview
The discovery dispute in this decision involved the City of Seattle's request to have defendant Professional Basketball Club, LLC (“PBC”) search for and produce responsive emails for six of its eight members. In January 2008, PBC produced approximately 150,000 emails from two members of PBC. It objected to producing emails for the remaining six PBC members because such a search would “increase the universe exponentially” and would generally produce irrelevant documents. The City moved to compel production of the requested emails.

The court first considered whether the requisite principal-agent relationship existed to establish that PBC had the legal right to obtain documents upon demand from its members. The court analyzed PBC’s operating agreement under Oklahoma law, and concluded that the City had met its burden in establishing that PBC had “possession, custody, or control” over the at-issue documents for purposes of Fed. R. Civ. P. 34(a).

Next, the court found that the substance of the requested emails – information regarding the formation of PBC and the Sonics' finances – may be relevant to the underlying issues. Given the liberal discovery rules, the court declined to limit the City's inquiry on relevancy grounds. It stated that, whether such evidence warrants admissibility is a distinctly separate question that would be addressed at the appropriate time.

Finally, the court observed that the Federal Rules contemplate a specific requirement when a party objects to the production of electronically stored information, citing Fed. R. Civ. P. 26(b)(2)(B). The court faulted PBC’s lack of specificity, stating that PBC had not explained why producing the emails at issue would be unnecessarily burdensome.

It continued: PBC also states in its moving papers that the emails add “nothing to the case except mountains of work for no return.” But a bald assertion that discovery will be burdensome is insufficient in light of Fed. R. Civ. P. 26(b) (2)(B). The Court is not permitted to presume the potential burdensome effects upon a party. The parties have already agreed upon a group of search terms that PBC previously used to search Messrs. Bennett and McClendon's emails and the Court assumes those terms may be used again to make further searches efficient.
Thus, the court granted the City’s motion and ordered PBC to produce the emails of the remaining PBC members within two weeks of the order.

The City of Seattle is represented in the case by K&L Gates attorneys Jeffrey C. Johnson, Paul J. Lawrence, Thomas Slade Gorton, III, Jonathan H. Harrison, and Michelle D. Jensen, along with Thomas Aquinas Carr and Gregory Colin Narver of the Seattle City Attorney's Office.

Labels: 26(b) (2)(B), 26(f), Discovery, FRCP, Meet and Confer, Search Terms

posted by Charles Skamser @ 2:12 PM 0 Comments