The Conceptual Search Game is Finally On!!

I have been writing (probably preaching to some) about advanced search technology, the differences between conceptual search and keyword search and the importance of advanced search technology in both the Early Case Assessment (ECA) and document review phases of eDiscovery for the past 3 years.

Concept Search Cash Law Emerging http://ediscoveryconsulting.blogspot.com/2008/06/concept-search-case-law-emerging.html
Concept Search vs. Keyword Search http://ediscoveryconsulting.blogspot.com/2008/12/concept-search-vs-keyword-search-in.html
Litigators Need ESI Analytics – Not Boolean Search Tools http://ediscoveryconsulting.blogspot.com/2010/05/litigators-need-esi-analytics-not.html

However, I have been somewhat disappointed in regards to the level of adoption of true conceptual search technology by the leading Litigation Technology vendors.  That appears to be changing.  As an example, in a June 17, 2010 post by StoredIQ titled, “Email Search: Nowhere to Hide”, the author provides an overview of StoredIQ’s search technology with specific focus on Natural Language Processing (NLP).  The reason that I am pointing this out is that 12 months ago, StoredIQ would not have been spending marketing dollars or Blog space on Natural Language Processing (NLP) because the market didn’t know what it was and didn’t care.

In this Blog post, StoredIQ now contends, “Probably of greatest interest to litigators during the discovery process is StoredIQ’s ability to perform natural language processing (NLP), which is the ability to extract linguistically derived natural language concepts from within email and user files including people, places and things. Legal teams can immediately search using over 250 out-of-the-box concepts and attributes including credit card accounts, social security numbers and stock symbols. NLP identifies word usage based upon context within a sentence. For example, NLP can identify if the word ‘will’ is used to identify a person’s name, a legal document or an auxiliary verb showing intent. StoredIQ has proprietary technology for adaptive sentence boundary disambiguation (ASBD) which substantially increases the precision of Natural Language Processing to address common grammatical deficiencies that are present in many business documents. No other information management technologies have this capability. NLP is a critical capability necessary to accurately perform eDiscovery, records management or risk management as full text indexes alone cannot provide the required level of precision.

Interestingly enough,  in my discussions with General Counsel and their litigation support teams from the Information Technology departments over the past 6 months, I have found a new awareness and appreciation for true conceptual search or semantic search or NLP.    So, StoredIQ is on the right track with their current product  offerings and I would bet that they have a product roadmap with more of the same.

So, I guess the conceptual search game is on and the other litigation technology vendors had better take notice of what their clients are saying in regards to what search technology they need.

The full text of the  StoredIQ Blog post is as follows:

A recent article by Jacob Goldstein, 23 Things Not To Write In An Email, illustrates the type of granularity as well as breadth of keywords that can be used by litigators during the legal discovery process to search for relevant information. He points out some keywords that may raise a legal red flag and should be used carefully when constructing emails. However, today’s technology search capabilities provide such precise, complete and accurate results, that there just isn’t anywhere to hide.
For instance, StoredIQ’s advanced search capabilities can look within compressed files, email archives and email attachments, in addition to the text contained in the email message itself. It can also search non-printable text within a document or email and can search through comments and revisions. In addition to search using keywords, StoredIQ supports many advanced search capabilities including:

• Single term search
• Multiple term search
• Concept-based search
• Boolean operators
• Logical grouping of terms
• Wildcards within search terms or Boolean expressions
• Proximity searches
• Natural language entities
• Regular expressions
• Macro-based searches
• Object level attributes
• By hash value (digital signatures)

Probably of greatest interest to litigators during the discovery process is StoredIQ’s ability to perform natural language processing (NLP), which is the ability to extract linguistically derived natural language concepts from within email and user files including people, places and things. Legal teams can immediately search using over 250 out-of-the-box concepts and attributes including credit card accounts, social security numbers and stock symbols. NLP identifies word usage based upon context within a sentence. For example, NLP can identify if the word ‘will’ is used to identify a person’s name, a legal document or an auxiliary verb showing intent. StoredIQ has proprietary technology for adaptive sentence boundary disambiguation (ASBD) which substantially increases the precision of Natural Language Processing to address common grammatical deficiencies that are present in many business documents. No other information management technologies have this capability. NLP is a critical capability necessary to accurately perform eDiscovery, records management or risk management as full text indexes alone cannot provide the required level of precision.
I know a lot of these terms can be a mouthful, but the underlying take away is that legal teams have the technology to precisely and accurately search electronic data, including email, making it much easier for litigators to discover data that was at one time hidden from them.

Labels: Conceptual Search, Early Case Assessment, ECA, eDiscovery, email, Keyword Search, Natural Language Processing, Seach Technology, Search Terms, Semantic Search, StoredIQ

posted by Charles Skamser @ 11:59 AM 0 Comments

Litigators Need ESI Analytics – Not Boolean Search Tools

This morning as I was enjoying my Monday morning coffee and reviewing the latest “Blog postings and press releases” in eDiscovery, I came across an article highlighting a research study published by AIIM on March 29, 2010, titled, “Users Need Content Research Tools, Not Basic Search Tools” that indicated users are more interested in finding good content analysis tools as opposed to just search tool. It is interesting that this study was not specifically written about eDiscovery. However, I would suspect that if AIIM had polled just eDiscovery professionals that they would have gotten an even louder call (more than 70%) for good eDiscovery analytics.

eDiscovery technology vendors are making tremendous strides in regards to enabling users to conduct some analytics during the Early Case Assessment (ECA) phase and even during the Document Review phase of the EDRM. However, I contend that most searches that are done today are keyword searches based upon a list of keywords that were produced by outside counsel (probably Associates and paralegals) with little or nor real knowledge of the Electronically Stored Information (ESI).

This archaic and potentially dangerous practice is no doubt a huge step forward from sticky notes, yellow pads and Excel spreadsheets. However, with the advanced analytical tools that have been on the market for general business analysis for years, there is no excuse for eDiscovery professionals to not be using advanced analytic search technology such as conceptual search for eDiscovery.

I have written extensively about this topic on this Blog. Following are links to some of those posts:

Become eDiscovery Superheroes: http://ediscoveryconsulting.blogspot.com/2010/03/become-ediscovery-superhero-with.html

Concept Search vs. Keyword Search in eDiscovery: http://ediscoveryconsulting.blogspot.com/2008/12/concept-search-vs-keyword-search-in.html

The Fog is Lifting on Conceptual Search in eDiscovery: http://ediscoveryconsulting.blogspot.com/2008/12/fog-is-lifting-on-concept-search-in.html

Conceptual Search Case Law Emerging: http://ediscoveryconsulting.blogspot.com/2008/06/concept-search-case-law-emerging.html

Labels: Conceptual Search, eDiscovery, Keyword Search

posted by Charles Skamser @ 11:04 AM 0 Comments

Become an eDiscovery Superhero with Conceptual Search and Categorization Technology

I grew up watching Batman and Robin on a really small black and white TV and reading Superman comic books. And, I have really enjoyed the resurgence of the Batman movie series. Given this fondness for superheros, I wanted to post a recent paper entitled, "Become an ESI Superhero" by Herbert L. Roitblat, Ph.D. of OrcaTec LLC and eDSG.

Dr. Roitblat takes a really clever approach to explaining eDiscovery categorization technology in terms that we can all understand. He contends that its time for us all to put on our tights and join the Justice League of Superheros. I'm not sure that the world is ready to see some of us in tights. But, Dr. Roitblat's analogy is very helpful in understanding how these new technologies can be used to appear as a superhero.

He contends that, "Categorization can help to make the review much more reliable by using the machine to learn the decision patterns of an expert and then using those decisions as recommendations for more detailed review. Think of it as a form of Vulcan mind meld that transfers the expertise of the best expert available to the rest of the review staff. "

He states that, "Semantic clusters help to gain a quick overview of what the collection is about and also suggest key terms and phrases that can be used to identify responsive documents. They provide a quick method of determining which documents merit further immediate review and which can be safely set aside. After review, if some documents in a cluster are marked responsive and others are not, it may be useful to examine why not."

And, he concludes that. "Concept selection gives you x-ray vision into the meaning of your collection. It lets you identify what the words mean in this particular context and to identify documents based on their meaning. It helps to highlight the documents that are most about a specific concept from the specific point of view of the context. For example, among the Enron emails, the word "osprey" is not used to refer to the bird, or the aircraft but to one of the off-books partnerships that got Enron into so much trouble."

If all of this really works as advertised, it will make us all look like superheros. Whether or not you decide to wear the tights is up to you.

The full text of Dr. Roitblat's paper is as follows:

Super heroes have super powers. Their powers enable them to do things far beyond the capabilities of ordinary people. Superman had his X-ray vision. Wonder Woman had a lariat that compelled complete honesty. The Flash had super speed. Batman had his superior intellect and technology.

These members of the Justice League each had powers that would come in handy managing today's eDiscovery. If these powers were available to lawyers today, would you use them?

How would or should lawyers go about making decisions about using their super powers? There are legal issues, certainly. But as far as I can see, the really critical question is whether these powers provide capabilities beyond those of traditional eDiscovery processes.

The eDiscovery powers I'm thinking about revolve around technology. We cannot afford to wait for a lightning bolt to strike. They include categorization, clustering, and concept searching. Tools like these have the power to help you see inside the case materials, derive the honest information from them with super speed, and amplify your superior intellect.

The goal for the eDiscovery process is to identify the ESI that is potentially relevant and to separate it from the ESI that is not. Ten years ago, we could argue about paper versus plastic, about whether it was more or less efficient to review documents on a computer screen or on paper. At the time, I worked on a case involving 13 million pages, which the producing party was determined to produce on paper. That would have been about 65 tons of paper, the weight of an old-fashioned steam locomotive. There was simply no way that the receiving party could go through all of that paper in a timely way. Just sorting the pages into date order was a daunting task.

Since that time, the volume of ESI that must be considered has only continued to leap over tall buildings. It is no longer practical to have the managing partner on a case read through all of the available documents. Few attorneys actually have the super power, the Flash's speed, for example, to read through millions of pages in a short time, so they resort to other means to help them get through it.

Categorization
One of the oldest approaches is to hire an army of temporary attorneys to read the documents. When Verizon was acquiring MCI in 2005, they responded to a DOJ second request by hiring 225 attorneys for four months, working 16 hours per day, 7 days per week. And all this effort was needed to review just 1.3 terabytes or 1.6 million documents. The review, alone, cost over $13 million or about $8.50 per document. As the volume of ESI continues to grow into multi-terabyte collections, this approach is simply not sustainable.

There is also evidence to suggest that this approach is not as accurate as it might be. What level of attention can a reviewer sustain while reading documents 16 hours a day, seven days a week, week after week? We had one client, in fact, in an unrelated matter, who asked us to use search technology to filter out the jokes because the reviewers were spending way too much time on them, rather than plowing through the potentially relevant material.

We (Roitblat, Kershaw, & Oot, 2010) recently published a paper in the Journal of the American Society for Information Science and Technology (JASIST) analyzing the performance of human reviewers in comparison to two computer-assisted review systems. None of the authors of that paper has any financial relationship with the reviewers or the companies providing the computer categorizers.

We set out to examine the idea that these computer systems could yield results that are comparable to those that would be obtained with a human review, and we found that they were. If we somehow knew which documents were truly responsive, then we could compare the judgments made during the first review with these true judgments. Unfortunately, no such oracle was available, so instead, we had to settle for a comparative method. We assessed the level of agreement between a new traditional review by new professional reviewers with the original review and between the computer systems and the original review. By comparing a new human review with the original review, we get an assessment of how well the traditional approach captures reliable aspects of the document collection. If the computer systems perform no worse, then it may be reasonable to use systems like this, rather than to spend the time and money needed to hire humans to do the work. I'll return to this assertion in a bit, after discussing some of the results.

After being trained on the issues involved, two teams of experienced professional reviewers were given a random sample of 5,000 documents to review for responsiveness. We could now assess the level of agreement of each of the teams with the original review and with each other. Team A agreed with the responsiveness decisions of the original review on about 76% the documents. Team B agreed with the original review on about 72% of the documents. They agreed with each other on about 70% of the documents.

You might think that the reason for such low agreement between the teams and the original review was because of the extreme conditions under which the original review was conducted. I have little doubt that the original review could have been done more reliably with more time (and more expense), but these results do not support that conclusion because the two re-review teams agreed with each other, even less often than they agreed with the original review, and were under practically no time pressure. Instead, it would seem that the low level of agreement is more likely explained by the unreliability of human responsiveness judgments in general. People are just not very reliable at identifying responsive documents, even in a small collection of a few thousand documents.

The two computer systems, on the other hand, agreed with the original review on about 83 % of the documents, certainly no worse than the level of agreement to be expected based on the human review . Replacing the army of human reviewers with a computer did not decrease the level of agreement, but it would very likely save a substantial amount of money.

Alan Turing, considered by many to be the father of computing, developed a test for assessing machine intelligence. His argument was that intelligence is a function. If the outputs of a machine are indistinguishable from the outputs of a human under a specific set of circumstances, then the computer could be said to implement the same function as the human—in this case the intelligence function.

In Turing's test all communication is done through a written medium, say a computer keyboard. The tester is supposed to have a conversation with a partner in another room. If the tester cannot tell whether she is communicating with a person or a machine, then the machine is said to be producing equivalent results and to be executing the same function. The computer could be said to be genuinely intelligent.

We could apply this same methodology in assessing computer versus human judgments of responsiveness. If we cannot tell the difference between the two kinds of systems, then we can conclude that they perform the same function. Based on the data reported in JASIST, that seems to be the case. If anything, the computers were a bit more reliable than the humans. If Batman were an attorney, he would use this kind of technology to gain an advantage over his adversaries. Think of it as a kind of BatReview.

Not every attorney is convinced by these results. For some, this skepticism reflects a romantic notion that there must be something special about having real live humans read every document. Their claim is that humans will find responsive documents that the computer will miss.

Although it may be true that humans will find documents that the computer might miss, it is at least equally true that one human will find documents that another human might miss and conversely, miss documents that another human reviewer might find responsive. It is also true, that the computer is likely to find responsive documents that the human might miss. The agreement between two groups of humans was no higher than the agreement between the computer systems and the original review. The available evidence does not support the claim that humans will find more responsive documents or that they will retrieve fewer nonresponsive documents than either of these categorization systems will.

Another claim is that the character of the documents missed by the computer will somehow be different from the character of the documents missed by the humans. This claim is more difficult to assess, because it is not obvious how to measure the character of the documents that are missed by one system or another. In order for this notion to be valid, however, and still allow the machines to achieve levels of agreement that are equal to or higher than those achieved by humans it would require that the humans miss an equal number of documents that the computer finds responsive. It implies that there is some systematic difference in the documents that the people find and the computer does not and some systematic difference in the documents that the computer finds and the people do not. It's not impossible that both systematic differences exist, but its practical significance is, at best, elusive.

How well any system, whether human or machine, performs is a matter of measurement. It seems unreasonable to claim that humans are somehow better than computers at distinguishing responsive from nonresponsive documents without measuring their performance along dimensions that matter. For example, the low level of agreement between human reviewers often comes as a shock to attorneys. Every review should include quality measures, whatever technology is used to perform it. Superman was not a superhero just because he flew around, that would only make him super. He was a superhero because he was successful at thwarting bad guys. His effectiveness could be measured by the number of evilness of the villians he defeated. The effectiveness of review should be measured by the ability to identify responsive documents and eliminate the nonresponsive.

Practically every process can be improved once it is measured. The steps taken to improve the quality of the review should be determined by the degree of risk in the case, in other words, they should be based on judgments of reasonableness.

The two computer systems used in the JASIST study did not make up their categorization by themselves. They don't actually decide what is responsive and what is not responsive. They form their categories on the basis of input from people. They implement the judgment of their "trainers" rather than make up their own. One system used in this test learns how to distinguish between responsive and nonresponsive documents from example judgments made by reviewers. The system is given a set of documents that the reviewers determined were responsive and a set that the reviewers determined were nonresponsive. From its analysis of these two sets, it derives a set of computational rules that distinguish between the two and applies these rules systematically to the remaining documents. This is the most common form of machine learning and automated categorization.

The other system was trained by linguists and attorneys to distinguish between responsive and nonresponsive documents. The trainers read the request and the training information provided to the original reviewers. They then adjusted the system's algorithms until it distinguished between documents in the way that these people determined was accurate. In both cases, the computer simply systematically implemented the decisions made by its team of trainers without getting bored, distracted, tired, or needing a vacation.

Keyword selection
Over the last few years, many attorneys have grown comfortable with one weak form of machine classification. Many of them have been using keywords to select or cull documents for further review. The attorneys pick a set of keywords or Boolean queries to use to select documents. These terms may be created by one side or negotiated between the two sides. Any document that contains one of these keywords or matches the query is selected for further processing, the others are simply ignored. For the most part, if a document does not have one of the keywords in it, it is never looked at again, so its information is effectively lost.

Keyword searching, although important, is the weakest form of machine classification available, hardly up to superhero standards, kind of like the Green Lantern with yellow things. The success of keyword searching depends critically on the ability of the attorneys to pick the right words that identify the responsive documents and do not overwhelm with nonresponsive ones. For more than 20 years we have known that attorneys are only about 20% successful at guessing the right words to search for (Blair and Maron, 1985). In the 2008 TREC Legal Track, they had two sides of an issue create search terms. The "defendant's" search terms retrieved just 4% of the responsive documents. The "plaintiff's" search terms retrieved 43% of the responsive documents, but 77% of the documents returned were nonresponsive, much higher than the 59% nonresponsive rate for the "defendant."

As you would expect, when they negotiated the search terms, as many of the thought-leaders in eDiscovery suggest, the results were intermediate, but still not stellar. Negotiated search retrieved only 24% of the responsive documents (http://trec.nist.gov/pubs/trec17/papers/LEGAL.OVERVIEW08.pdf, p. 10). In the 2007 TREC, it was even lower. Of the documents that were retrieved, 72% were found to be nonresponsive. Keyword culling serves to reduce the volume of documents that will be considered for review, but it does not do a very good job of identifying responsive documents. Cooperation may be the better policy, but it's no lariat of truth, unless further steps are taken.

One reason for the poor performance of these search terms might be that they were based on the attorney's expectations for the words that ought to be in the collection and ought to distinguish between responsive and nonresponsive documents, rather than for the words that actually were there. For example, people often misspell words in emails. A name like "Brian," for example, might be spelled "Brain," or "Bryan." "Believe" might be misspelled as "beleive." Document authors, especially email authors, may use nonstandard abbreviations. They can be very creative in the words they choose to use. There are over 200 synonyms, for example, for the "think." Did they "buy," "purchase," "acquire," or just "get" a new car, for example? Groups all have specialized jargon. Lawyers may have one way to talk about issues in the case, but the document authors rarely think or write like lawyers.

Identifying the right terms to search for can be helped by looking at the words that are actually in the ESI. One easy way to do this is to simply print out or display an index of the collection. This list is sometimes called a word wheel. Then, the parties can select terms from this list, rather than trying to make up a list from their imagination.

Clustering
A more powerful technique is to use semantic clustering to group documents with similar content. Using one of a number of techniques, the computer groups together similar documents and finds a word or a phrase that describes that group. A quick examination of these groups is often enough to determine whether the documents they contain are likely to be responsive or not. The labels can be used as potential search terms. Some of these clusters may be obviously responsive, some obviously nonresponsive and others may require looking at a few of the documents to figure out.

There is an added benefit if a given document can be in more than one cluster because documents can be about more than one topic. An email might say something like, "I'm bringing pizza to the party on Saturday, and by the way, the money that we stole is now in our Swiss bank account." If that email were clustered into only the pizza party cluster, no one would ever see it again and that information would be lost.

The clusters can also be used to select or cull. You can scan down a list of clusters and determine whether the documents in that cluster are likely to be responsive or not. Any document in a potentially responsive document could then be reviewed further, but documents that never appear in a responsive cluster can be set aside.

Here are a few of the cluster labels derived from a subset of the Enron data:

- ibm
- 2q
- bal month
- transactions
- software
- luis gasparini
- coaches
- generators
- lon ect
- hou ees
- el paso
- ebner daniel
- officials
- epmi long term northwest
- bonus
- game

If the issue involves financial information, then documents in 2q, epmi long term, transactions, and bonus clusters may be relevant. Terms like these would make good search terms, and you know that they are actually used in the documents and that there are enough of them to make up a cluster. Documents in the coaches, software, and game clusters are unlikely to be relevant. Because the same document can appear in more than one cluster, it will still be selected for review if it appears in at least one responsive cluster, no matter how many nonresponsive clusters it appears in.

Concept Selection
Concept searching is another tool that can help to amplify your powers. Concept searching identifies the meanings of words using any of a number of different technologies, including ontologies, thesauri, latent semantic indexing (LSI), and language modeling. Ontologies and thesauri are usually created by experts, who program word relations into the system. These knowledge engineers identify that the word "car" is related to the word "vehicle," for example. Systems based on one of these approaches contain only those relationships that have been explicitly programmed into them.

LSI and language modeling derive the meanings of words automatically from the context in which those words are used. These systems reflect the actual patterns of word use and are capable of discovering unanticipated relationships.

Concept searching works by expanding the query that the user submits to include the original term and additional, related terms. Concept searching identifies documents as relevant, even if the exact query term happens not to be in the document, because it searches not just for the original term submitted by the user, but for it and related terms. This approach tends to push the most relevant documents, the ones with the query term and the most context, to the top of the results list and to add a small number of related documents, which do not happen to have the query term, to the tail end. Concept search tends to mitigate the difficulty of guessing the right terms to search for, because it learns what terms are related to which. It helps to find documents based on their meaning, rather than solely on the presence of specific words.

Conclusion: Using These Superpowers
The eDiscovery superpowers described above can really help to reduce the cost and burden of eDiscovery. Categorization can help to make the review much more reliable by using the machine to learn the decision patterns of an expert and then using those decisions as recommendations for more detailed review. Think of it as a form of Vulcan mind meld that transfers the expertise of the best expert available to the rest of the review staff.

Documents identified by the categorizer can be reviewed first. It prioritizes processing so that the most likely documents can be completed first. It allows the reviewers to quickly gain exposure to the documents most likely to be relevant and learn from the examples what makes them relevant.

After the review, a sample of the decisions made by the human reviewers can be fed back to the categorizer and it can be retrained. If the reviewers were consistent in their review, then there should be few discrepancies between the categories assigned by the computer and the categories assigned by the reviewers. These discrepancies can then be examined to resolve those inconsistencies.

By examining the actual words used in the documents, the quality of keyword selection can be greatly improved. It then becomes easier to negotiate sensibly about which keywords to use. It becomes easier to demonstrate that you have met the requirement of the Federal Rules to conduct a reasonable search of the ESI. Most importantly, you increase your chances of finding the responsive documents without over-burdening the collection with irrelevant ones.

Semantic clusters help to gain a quick overview of what the collection is about and also suggest key terms and phrases that can be used to identify responsive documents. They provide a quick method of determining which documents merit further immediate review and which can be safely set aside. After review, if some documents in a cluster are marked responsive and others are not, it may be useful to examine why not.

Concept selection gives you x-ray vision into the meaning of your collection. It lets you identify what the words mean in this particular context and to identify documents based on their meaning. It helps to highlight the documents that are most about a specific concept from the specific point of view of the context. For example, among the Enron emails, the word "osprey" is not used to refer to the bird, or the aircraft but to one of the off-books partnerships that got Enron into so much trouble.

Armed with these superpowers you are now ready to combat your adversaries in the world of eDiscovery. Go put on your tights and join the Justice League.

Labels: Categorization, Clustering, Conceptual Search, eDiscovery, Keyword Search, Superheros, The Justice League

posted by Charles Skamser @ 3:00 PM 0 Comments

Web 3.0 in eDiscovery

Over the past year I have been following discussions about Web 3.0, the next generation of the Internet. And, even though my wife is yelling at me to get ready to go Christmas shopping, I wanted to get this Blog posting done today as I believe that Web 3.0 is going to have a profound effect on eDiscovery in 2010 and beyond. Not only in terms of what we have to "Discover" but also in terms of how we "Discover". And, I am going to use Christmas shopping in my example of how to use Web 3.0. I am not sure what that says about my priorities. However, here goes.

From a SciFi stand point, Web 3.0 moves us closer to a more "Hal 900" like computer experience. So, instead of asking your computer to help you "find information" by typing keywords into Google, you will ask your computer to help you "complete a more conceptual task" such as as last minute Christmas shopping for my family that includes a stop for lunch with burgers and very large HDTV to watch the Packers play the Steelers. To put this into perspective, experts in the world of search would describe this approach as maturing from keyword search to conceptual semantic based search. And, the process of storing data on the Web in a form that machines can naturally understand, or converting it to that form, is called the "Semantic Web", a phrase attributed to Tim Berners-Lee, inventor of the Web (it wasn't Al Gore) and Director of the World Wide Web Consortium (W3C).

Web 1.0 was basically just a static library of information that user could access and utilize much like checking a book out of a library. Web 2.0 (the current state of the Internet), enables users to actually change the content of pages with additional information effectively making 'Web pages" living pages that grow through interaction with visitors. Web 2.0 is also characterized by social networking such as facebook and Linkedin, mass sharing of information via application such as Twitter and access via devices beyond the traditional computer.

Berners-Lee disputes the existence of Web 2.0, calling it nothing more than meaningless jargon [source: Register]. Berners-Lee maintains that he intended the World Wide Web to do all the things that Web 2.0 is supposed to do. His vision of the future Web is similar to the concept of Web 3.0. Unfortunately, the Web's current structure of storing information has been designed to work best with the the keyword approach. And therefore, accept is a few isolated instances, Web 3.0 isn't really even a possibility in 2009. For the realities of Hal9000 interactive with the Web, there have to be some changes made to way information is stored and processed. Web 3.0 will have to have collections of information called ontologies, a file that defines the relationships among a group of terms.

Back to our Christmas shopping task, in order for Web 3.0 to be able to actually provide the information that I need to have a successful shopping trip, eat lunch and watch the game, Web 3.0 will have to have an ontology that understand my family, what they want for Christmas, the best places to get those gifts, restaurants that serve hamburgers that will also be Packer game, etc., etc. etc. Basically, much more information than is available today and more information that will be available anytime soon on the Web. However, this doesn't mean that the technology is not available. Semantic search research and development has been underway for many years and is actually already being used successfully in eDiscovery and other information management markets . It's use is just not very widespread and there is still a tremendous amount of confusion about how it works and the validity of the results (please note that I have written about conceptual/semantic search in eDiscovery on this blog several times).

So, what will eDiscovery look like in the world of Web 3.0. First of all, it won't be a digital transition (i.e. we won't wake up one morning with Web 3.0). It will be a more gradual analog transition with application providing the best ROI emerging first and the rest of the world following behind (i.e. classic early adaptors vs. laggers). And, as already indicated, eDiscovery is already the perfect arena for semantic search as users can isolate information, develop ontologies and then ask the Hal900 type questions like, "what relationships exist between these custodians that are not apparent?". Or, "tell me about potentially important relationships that may exist outside of the known custodian?". Eventually, as the ontologies and the yet unknown upstream next generation ontology type technologies emerge, new search technologies will be able to find just about any information that lawyers require to ensure they understand everything that they need to know to proceed with their matters with a very high level of confidence. And, more importantly, a much higher level of confidence that what could ever be provided by completing the same analytics manually.

Other markets that are also already benefiting from this technology are consumer marketing (uncovering buying patterns), anti-crime and anti-terrorism and drug research (find unknown drug interations), just to name a few.

What is even of more interest to me is how our laws will evolve in this new world under Web 3.0. Just as Hal900 will enable law enforcement and the legal system to better understand and analyze information, so to will this same technology enable criminal elements to become more sophisticated in their approaches. I could very easily envision scenarios where Hal900 is asked the question about the most efficient and least risky way to commit a crime.

It should all be very ineresting and exciting to watch.

Labels: Conceptual Search, eDiscovery 3.0, Ontology, Semantic Search, Web 3.0

posted by Charles Skamser @ 10:24 AM 1 Comments

Early Case Assessment (ECA) Should be Emerging as a Best Practice

This question was addressed in an article/marketing piece published on November 2, 2009 on the The Metropolitan Coprorate Counsel website by Rich Cohen and Joe Garber titled, "The Evolution Of Early Case Assessment From Simply A Good Idea To An Enterprise Solution".

And, even though the article is a bit self serving from the standpoint that they are peddling Vestigate, the ECA platform that their company, RenewData offers, Cohen and Graber actually do a really great job of outlining the issues and explaining some of the "real" value intrinsic in the ECA model.

There is no doubt that pushed and proded by the changes to the Federal Rules of Civil Procedure (FRCP) and more specifically the requirements of Rule 26(f), most litigators now understand the "concpet" that knowing where your ESI is located, collecting and analyzing that ESI and then performing an intelligent culling of that ESI down to the truly relevant and possibly responsive ESI is a fundemental key to realizing success in the eDiscovery process. And, that it can be a key competitive advantage in preparing for the initial meet and confer meeting(s).

However, and as pointed out by Cohen and Graber, the technology platforms to support this ECA process is all very new and therefore, for the ill prepared or misinfomred, can out to be a very technical endeavor with less than understandable and/or even acceptable results. In addition, eDiscovery vendors, whether intentional or not, have done an extremely good job of completely confusing the marketplace in regards what all of this new technology does based upon conceptual search vs. the old EDD keyword search approach and and how to use it.

I have addressed some of these issues in past Blog Posts:

The eDiscovery Paradigm Shift: Concept Search vs. Keyword Search

The eDiscovery Paradigm Shift: Concept Search Case Law Emerging

And, I plan to post more of the same over the remainder of 2009 and into early 2010 and have some great overview of very interesting new ECA and Conceptual Search technology. However, for the purposes of this post, I would like to make the following observations regrading ECA and Conceptual Search Technology:

1. Early Case Assessment (ECA) technology is maturing and does have the potential to be somewhat of a silver bullet for eDiscovery market.
2. Any Early Case Assessment (ECA) solution that wants to rise to the top of the vendor pile is going to have a very sophisticated "true" conceptual search solution possibly sitting in a technology stack with Lucene and associated Open Source solutions that have been adopted into the commercial markets. However, next generation conceptual search and associated support for next generation ECA such as the eDiscovery Tool Kit from Orcatec, can be farily easy to integrate with older solutions as it doesn't necessarily have to sit on any specific newer technology stack.
3. As a follow on to # 2, don't get confused by vendors that are contending that they have conceptual search when in fact that have "old" boolean key word search. There is a big difference (see my posts on this topic).
4. The true sliver bullet ECA solution will be more of an end-to-end eDiscovery solution that will be SaaS based (i.e. users will be able to access it over the Web), will enable ingestion of 400+ files types, will be able to normalize all embedded files, will be able to apply very sophisticated analytics based upon conceptual search and related technology, will support iterrative document review of both the native and image versions of documents (as opposed to linear review), will enable user extensible tagging, will include workflow management and stakeholder dashboards to track project status, will support production into the standard required formats, and; may even include new automated/machine driven review (stay tuned for more posts on this topic in the comming weeks).
5. Courts are beginning to undertsand how all of this works.
6. Don't be affraid to bring in the experts to help with understanding all of this. And, the technology vendors may not necessarily be the experts as they don't necessarily understand ECA best practices, etc.

So, Early Case Assessemnt (ECA) is evolving and growing and trying to find its way. But, don't wait too long to jump on the train as it is leaving the station and once gone, you may be left without a way to complete the 26(f) requirements and may be at a substantial disadvantage to your opponent.
The full text of the articel by Rich Cohen and Joe Garber is as follows:

Ask 10 people who are "in the know" about managing electronic data for legal or regulatory purposes, and I suspect that you wouldn't find one who says that Early Case Assessment (ECA) isn't a great concept. The idea of leveraging advanced technology to help you be smarter about how you approach eDiscovery and save millions of dollars in downstream legal review would seem to be a slam dunk. Yet despite the obvious benefits, the adoption of ECA technology has been slow to rise.

The reasons for this apparent anomaly are actually threefold: 1) despite a great amount of buzz on the topic a clear definition of ECA has yet to emerge, causing many organizations to remain hesitant about deployment; 2) the software that has been introduced to the market so far has been somewhat misaligned with the fundamental needs of its users; and 3) a simple approach to evaluate the various technology options for this process has been unavailable to date.

This article will address these issues head on. In the process, we will clear up any confusion about what ECA is, identify the gaps in how this technology has been delivered thus far, provide a new framework for evaluating different solutions available, and ultimately tell you about a unique approach to ECA that is much more in line with the needs of the market.

What Is ECA?

Today, many organizations are finally starting to get their arms around their total costs of eDiscovery, and they're finding that the primary cost driver is attorney review. To combat excessive spending in this area, many smart organizations are looking for ways to analyze their data further upstream in the eDiscovery process so they can make some strategic decisions before involving large teams of lawyers that they pay by the hour.

To that end, ECA tools are designed to quickly provide a snapshot of what is contained within all your data and help you make sense of it all. Knowing where all your information is, and what risk exposure your organization has right up front will help you decide if you want to settleor if you intend to fight the case, ECA may help you determine how you will staff the case and what approach you'll take as a defense. Perhaps just as important, it can help you set aside clearly non-responsive data so you don't have to pay to review it later on.

ECA is attractive to organizations in virtually all industries, particularly those that have a high volume of legal or regulatory actions who are facing pressures from budget cuts and/or downsized legal and IT staffs. It is contrasted with traditional review technology because it takes place much earlier in the eDiscovery process, is generally conducted by a more senior-level attorney (who has the experience to make the strategic decisions), and provides a higher level of analysis than that which often takes place in review software to prepare your data for use in court.

The ECA Challenge (So Far)

Again, despite the obvious benefits, many organizations have been slow to adopt ECA to this point. There are several reasons for this, including internal cultural issues (e.g., people have a hard time adapting to change), and that ECA technology was a bit in its infancy until recently.

But possibly a bigger barrier to adoption has been how this technology has historically been delivered. ECA came to pass when some of the larger software companies figured out that the conceptual search technology they'd invested in over the past few years had some clear overlap with the need to analyze a great amount of data, very quickly, for legal reasons. This software, which uses some complex mathematics to find patterns in data, could help you more quickly categorize the data you were looking at and thus expedite your search.

However, what was taking place under the hood was difficult to understand - and more importantly, difficult to explain to opposing counsel and the courts. As a result, many felt they really couldn't re-use their work product in the subsequent review stage. That meant users had to use one technology for ECA, and then start all over with their search efforts in a separate review application they could better explain to all parties later on. Naturally, using one application for ECA and a second for review led to more risk and cost than most organizations were comfortable with.

The other drawback to how ECA technology was traditionally being delivered was that you didn't have a great sense that you had really seen "everything that needed to be seen." It's no surprise by now that ECA is about looking at only a small subset of your overall data to make some strategic decisions - and most technologies built for ECA do not have the tools in place to provide confidence to its users that they haven't missed a critical piece of evidence as they did their quick analysis.

The Market Evolution

As we set our sights on a new decade, ECA technology is poised to make a leap from simply a good idea to being a true enterprise solution that can deliver on the promises made when the technology was originally conceived. This is because a small handful of providers have arrived at the inevitable conclusion. In order to deliver software that is widely adopted in the market, a number of changes have to be made to how it is packaged and delivered.

The new market mandate is for ECA technology that is able to provide a quick snapshot of an organization's data, can be leveraged for both ECA and review, is simple enough to be used by the senior-level attorneys who generally conduct this work, and provides users with measureable confidence that the data they've based their strategic decisions on is truly representative of the entire corpus.

Translated more simply, the ECA technology that will provide the greatest value to organizations subject to legal or regulatory action will be those applications that cast a wide net on all your data, are simple to use, are highly transparent (i.e., do not rely on black-box search), are integrated into an end-to-end eDiscovery solution, and can deliver measureable certainty that you've seen what needs to be seen.

While simple in concept, this is a tall order that very few technology vendors will be able to fulfill going forward.

A New Breed Of ECA Technology

One company that has made great strides in delivering a market-driven ECA solution is RenewData, an eDiscovery, archiving, and information governance service provider. Its Vestigate software is based on a methodology that was being used with an exclusive set of clients and which has just recently been translated to a technology-based offering for the mass market. This Software as a Service (SaaS) solution lets humans take back control of their data, and is ahead of the curve in terms of delivering the value that organizations are now demanding.

There are three key components of the software that accelerate access to relevant documents and provide measurable accuracy to ensure that important documents have not been missed:

1. The first key component is a simple user interface that allows a small team to quickly identify what's relevant in a given document just by using a mouse to highlight critical language (or metadata) that's relevant to the case.

2. The system then finds all similar documents using nothing more sophisticated than Boolean searching, and automatically bulk tags those documents to eliminate the need to read significant amounts of redundant data. This happens until it appears that all potentially relevant documents have been identified.

3. The final key aspect of Vestigate is statistical sampling for quality control. This important analysis is put in place to provide confidence that you have taken reasonable steps to locate and analyze the most critical data. This process, based on zero-defect testing that is widely used in many industries (such as pharmaceutical drug testing), delivers a measurable confidence levelup to 99.9 percent certainty that all relevant documents have been identified.

Most Vestigate clients start out with an 85 percent confidence level that the critical data has been analyzed during the ECA stage, which is much higher than humans are capable of achieving on their own with redundant tasks, according to a wide range of published reports. After reading only five to six percent of the total data in question, organizations have a very good sense of what's contained within their data and have their most senior-level attorneys interact with the system (regardless of their comfort with technology) to make the key strategic decisions for a given case.

The ease-of-use and measureable certainty are two clear differentiators for Vestigate, but another is its integration in an end-to-end eDiscovery solution and its ability to be leveraged for both ECA and review. If the senior attorneys decide that they intend to fight a case after their analysis, they can then go back to the data in Vestigate and keep working. Using the exact same tool, they can proceed with first-pass review and have a very small team mark data for relevance. Generally, using the same text-highlighting, bulk-tagging, and quality-control capabilities, the team will only review about 15 percent of the overall dataset, achieve up to a 99.9 percent confidence level that they've reviewed all potentially relevant data, and can ultimately save the organization up to 70 percent in overall review costs.

Summary

ECA is a hot topic in eDiscovery circles (and boardrooms) today for a good reason. The cost of attorney review is extremely high, and it's only destined to increase as the volume of data expands. A number of technologies are available to help organizations bring these costs in line.

But buyer beware - not all ECA solutions being sold in the market today are the same. In fact, there is a wide range of offerings available, many of which have yet to evolve from concept to solution. Make sure you understand what each software provider is delivering and look for a solution that allows you to get a good sense of what your data is telling you about the case at hand - without forfeiting overall cost savings or certainty that you've seen what needs to be seen before you make your decisions.

Labels: Conceptual Search, Early Case Assessment, ECA, RenewData

posted by Charles Skamser @ 10:36 AM 1 Comments

The New Generation of eDiscovery Search

The New Generation of eDiscovery Search technology train is getting ready to leave the station. However, after walking the tradeshow floors and attending many of the breakout sessions at last weeks LegalTech in New York, it is obvious that there is a tremendous amount of confusion regarding the definition and scope of the New Generation of eDiscovery Search technology and more importantly, how the courts view the use of such technology.

With the accelerating volume of Electronically Stored Information (ESI) or what I like to call Electronically Stored Evidence (ESE), the current legacy search technologies built into the current legacy eDiscovery tools and the associated best practices for document review are beginning to have a hard time "keeping up". Further, there is a tremendous amount of confusion and trepidation among litigators in regards to potential malpractice claims, sanctions and adherence to Rule 702 and Daubert challenges associated with employing the New Generation of eDiscovery Search technology. Finally, litigation technology vendors, whether purposely or not, have confused the market with fancy new marketing terms like "conceptual search", "transparent search", "linguistic search" and "clustering" without any real explanation of how they work and how to correctly employ them. Therefore, I thought that it was time to restart the campaign to both educate and lobby the eDiscovery industry in regards to the New Generation of eDiscovery Search.

First of all, I want to start with education regarding the pertinent issues. Without a doubt, one of the best article posted over the past 12 months on the legals issues surrounding the topic of what I am calling the New Generation of eDiscovery Search, was written by By Wayne C. Matus and John E. Davis in the New York Law Journal on October 31, 2008, titled, "Do Your Searches Pass Judicial Scrutiny?"

Since it is my impression that many of these very important issues are still either unknown to most in the eDiscovery "business" or are being ignored, I contacted Mr. Matus this week to get permission to repost his article.

Following is the full text of of "Do Your Searches Pass Judicial Scrutiny?":
Electronically stored information is increasing exponentially, and bills from law firms and discovery vendors to deal with this vast sea of data escalate significantly each year. Jason Baron, the director of litigation at the National Archives and Records Administration, believes that ESI is growing so fast that even with unlimited funds and human resources it will soon be impossible for humans to review these large document populations.[FOOTNOTE 1] Still, lawyers faced with potential malpractice claims and sanctions are loath to try new methods for handling the problem. It is time for change.

The traditional means used by litigators to address ESI is the application of keywords and Boolean search terms to identify relevant and non-privileged materials.[FOOTNOTE 2] While acknowledging that this method is unquestionably deficient, a recent article published in this publication concluded that "the available evidence suggests that keyword and Boolean searches remain the state of the art and the most appropriate search technology for most cases."[FOOTNOTE 3] We agree that, in a perfect world, if the parties can nevertheless meet and confer, and agree upon keywords to reduce the population to manageable proportions, the traditional judgmental method can be made to work. However, this is an imperfect world where plaintiffs and defendants do not always agree, and are not always equally motivated, to reduce costs. In fact, it is often quite the opposite. Moreover, even where the sides use judgmental sampling to agree upon keywords, the costs nevertheless usually remain too high.

THE JUDGMENTAL APPROACH
The judgmental approach to keywords ultimately fails because of "recall" and "precision." "Recall" measures how completely a process captures target data. "Precision" measures efficiency - the amount of irrelevant data captured along with the target data. Keywords, as judgmentally used by lawyers, recall too little, while capturing much that is irrelevant. An early landmark empirical study by David Blair and M.E. Maron[FOOTNOTE 4] showed that while lawyers thought they were retrieving about 75 percent of the relevant data, the true results were more like 20 percent. A subsequent study, conducted by the Text Retrieval Conference,[FOOTNOTE 5] confirmed this result, finding that only 22 percent of relevant documents were recalled using keyword search techniques, as opposed to approximately 78 percent found by other search techniques.[FOOTNOTE 6] Many lawyers will also tell you that it is common for reviewers to find only 10 to 40 percent of the recalled documents to be relevant, meaning lawyers are reading mostly junk.
We advocate two different approaches to yield better and more efficient results. First, we suggest that keywords are best used coupled with statistical, rather than judgmental, sampling. Second, we suggest that experienced counsel and vendors working with a combination of advanced conceptual search techniques can more efficiently and effectively deal with large amounts of ESI, resulting in a narrowed and enriched review set with a concomitant reduction in lawyer hours.

KEYWORDS DONE RIGHT
In Victor Stanley v. Creative Pipe, 250 FRD 251 (D. Md. 2008), Chief Magistrate Judge Paul W. Grimm of the U.S. District Court for the District of Maryland found counsel had waived the attorney-client privilege as to 165 inadvertently produced documents -- despite the use of 70 separate keyword searches in conducting their privilege screen -- because, among other reasons, counsel had failed to conduct "quality assurance testing." Clearly, judgmental sampling did not pass judicial scrutiny, while statistical sampling would likely have.

Counsel seeking to conduct a proper keyword search should instead consider the following steps:

• Sample the data. Counsel should isolate a random and statistically significant sample of the relevant datasets and then conduct a manual review of such data for relevance and privilege.[FOOTNOTE 7] This will educate counsel as to what to expect from the larger population and help in formulating keywords.
• Analyze and rank keywords. Counsel should then create and run search terms against the sample set, and (based on the information derived from the sample review) analyze their effectiveness by "recall" and "precision." This preliminary knowledge of the contents and richness of particular datasets will provide the basis to predict retrieval and review costs, and whether, for example, counsel should conduct any or just a limited review of such data.[FOOTNOTE 8]
• Review and repeat until satisfied that the search plan is defensible. This approach is plainly iterative in nature; it is the rare set of searches that achieves acceptable returns without adjustment. Successive application and fine-tuning of terms should permit counsel to achieve defensible levels of recall with superior precision rates. Practitioners should take note: One of the main factors cited by the court in Victor Stanley to determine if a party has conducted a reasonable search is if the party has reviewed a sample of the results to "assess its reliability, appropriateness for the task, and the quality of its implementation."[FOOTNOTE 9]

There is no consensus as to what percentage of recall will pass muster. Instead, counsel must be able to explain to the court the "reasonableness" under the circumstances of each step of the process, including the point at which a party was satisfied with the effectiveness of its search terms.[FOOTNOTE 10]

KEYWORDS: TO DISCLOSE OR NOT
Search terms created by counsel are generally protected, at least initially, by the attorney work-product doctrine, as their "mental impressions, conclusions, opinions, or legal theories ... concerning the litigation."[FOOTNOTE 11] But how does one show "reasonableness" of the search methodology without disclosing the keywords? Three recent cases, Victor Stanley, O'Keefe and Equity Analytics,[FOOTNOTE 12] have required that attorneys be able to explain and defend to the court, at its request, the methodology used to employ the search terms. One court has indicated it might find a waiver of privilege and require disclosure of the search terms.[FOOTNOTE 13]

OTHER FILTERING TECHNOLOGIES
Magistrate Judge John M. Facciola of the U.S. District Court for the District of Washington, D.C., recently pointed to authority that "concept searching" applications -- which use statistical and linguistic models to search for ideas as well as words and impose order upon disparate documents -- are "more efficient and more likely to produce comprehensive results" than keyword or Boolean searches.[FOOTNOTE 14] For example, the TREC 2007 Legal Track study found that 78 percent of relevant documents in a dataset were not found by Boolean keyword searches, but only by alternative search techniques.[FOOTNOTE 15] It is little wonder that Magistrate Judge Grimm has expressed optimism that concept-based searches studied by TREC would supplant keywords as the preferred method "for a variety of ESI discovery tasks."[FOOTNOTE 16]

These findings appear to have been confirmed. Earlier this year, the eDiscovery Institute disclosed its preliminary assessment of the study it conducted on the performance of computerized document review against human review. The study was conducted against a dataset drawn from the Verizon-MCI merger consisting of 1.3 terabytes and over two million documents. They concluded that computer systems allowed a comparable level of performance to be achieved with fewer people, less time and lower cost. While actual cost of traditional review was over $13.5 million, computer-assisted review was projected to cost just a fraction of that amount.[FOOTNOTE 17]

Concept search methodologies fall into three basic (and sometimes overlapping) categories:

• Probabilistic. This technique relies upon probabilistic search models such as "Bayesian classifiers," which evaluate and classify documents based on the interrelationships, proximity and frequency of usage of words found therein. The model may be given a "head start" by a sample set of relevant documents developed by attorneys at the outset of the process, which the computer analyzes and applies to the remaining documents. This technique can order groups and documents based on perceived potential importance to assist in the review process.
• Rule-based (or "clustering"). This statistically driven process analyzes the prevalence of words in documents and, based on such analysis, groups together documents interpreted as featuring like concepts. This technique can order documents by perceived potential importance as well.
• Linguistic. Sometimes referenced as "fuzzy search models," this technique seeks documents containing all forms of a target word or its synonyms in a general and/or case-specific thesaurus. Linguistic approaches may also rely upon statistics to analyze documents for terms along the same subject lines -- or sometimes to identify documents that use different ways to make the same point.[FOOTNOTE 18]

Differing tools often produce differing results, but some combination of each of these approaches (as well as Boolean keyword searches) -- using a transparent, iterative and measured process as described above -- can be used to best effect. As successive waves of ESI are received (as is often the case), moreover, certain of the concept-searching applications "learn" and become better at identifying correlations, associating documents with particular attributes with concepts of interest to counsel and minimizing false positives. Further, the statistics generated by this process permit counsel to draw educated lines as to where review should proceed and, sometimes more importantly, when it is reasonable to stop. The advantages of these powerful, computerized techniques become even more apparent where ESI reaches the terabyte range and the steep recall/precision tradeoff exhibited by keyword analyses may reach unacceptable levels. Two jurists have indicated in opinions an interest in hearing from experts as to such new approaches.[FOOTNOTE 19]

CONCLUSION
Given escalating volumes of ESI, with no end in sight, and the general impatience of courts with e-discovery mistakes, counsel and their clients soon may have no choice but to adopt discovery tools that are more efficient and precise than traditional Boolean search techniques. Courts have already put practitioners on notice of this emerging obligation. Combining measured approaches to search methodologies with advanced techniques can greatly assist in the organization of ESI and the cost-effective conduct of litigations and investigations. The future is now for these state-of-the-art search techniques.
Wayne C. Matus is a litigation partner in the New York office of Pillsbury Winthrop Shaw Pittman and one of two national leaders of the firm's e-discovery practice. John E. Davis is a senior associate in the firm's New York office specializing in e-discovery. Sandra Barragan, an associate at the firm, assisted in the preparation of this article.

::::FOOTNOTES::::
FN1 "EDD Showcase: Discovery Overload," Law Technology News, January 2008.
FN2 Although keyword searches and Boolean term searches are undeniably distinct, for purposes of this article we will refer to them interchangeably.
FN3 See "Assessing Alternative Search Methodologies," H. Christopher Boehning and Daniel J. Toal (NYLJ, April 22, 2008).
FN4 "An Evaluation of Retrieval Effectiveness for a Full-Text Document Retrieval System," Communications of the Association for Computing Machinery at 289-99, March 1985.
FN5 TREC is sponsored by the National Institute of Standards and Technology (NIST) and the Advanced Research and Development Activity of the Department of Defense.
FN6 These were the results of TREC 2007, the second year of the Legal Track study. See Jason R. Baron, Douglas W. Oard, Paul Thompson & Stephen Tomlinson, Overview of the TREC-2007 Legal Track, at §6 (linked at http://trec-legal.umiacs.umd.edu/). In a prior TREC-6 Ad Hoc Task study, for keywords to achieve just 50 percent recall, the architects had to accept a dismal 20 percent precision rate (whereby four of every five documents selected by keywords were nonresponsive). See H5 White Paper, Concept Search: Perceived Security, Actual Risk, at 2, citing Voorhees, Ellen M., and Harman, Donna, Overview of the Sixth Text REtrieval Conference (TREC-6), in NIST Special Publication 500-240: The Sixth Text REtrieval Conference (TREC 6), ed. E.M. Voorhees and D.K. Harman, 1-24 (Gaithersburg, MD: NIST 1997), and Voorhees, Ellen M., and Harman, Donna, Overview of the Seventh Text REtrieval Conference (TREC 7), ed. E.M. Voorhees and D. K. Harman, 1-24 (Gaithersburg, MD: NIST 1998).
FN7 E.g., Treppel v. Biovail Corp., 233 FRD 363, 374 (SDNY 2006).
FN8 See McPeek v. Ashcroft, 212 FRD 33, 35 (D. D.C. 2003) (ordering sampling of backup tapes to determine whether they contained relevant documents); Wiginton v. DB Richard Ellis Inc., 229 FRD 568, 570 (N.D. Ill. 2004) (ordering sampling of archived material based on keywords to determine whether it should be restored); see also Victor Stanley, 250 FRD at 261, citing The Sedona Conference Best Practices Commentary on the Use of Search & Information Retrieval Methods in E-Discovery, 8 Sedona Conf. J. 189 (2007) [hereinafter, "The Sedona Best Practices"]. For example, the sampling process may reveal that certain data sources (such as local hard drives) or file types (such as Microsoft Access files) have such low yield that the collection and review effort is not worthwhile. While opposing counsel may not agree to such decision, the data provided by the sample review will provide the evidentiary support needed to defend the reasonableness of such steps to the court.
FN9 Victor Stanley, 250 FRD at 256.
FN10 See, e.g., Security Financial Life Insurance Company v. Dept. of Treasury, 2005 WL 839543, *4 (D. D.C. April 12, 2005) ("In deciding whether an agency's document search is adequate, the issue is not whether other responsive records might possibly exist, but whether the search was adequate, judged by a reasonableness standard.") (internal citations omitted); see also Victor Stanley, 250 FRD at 261 n.10 ("the cost-benefit balancing factors of [FRCP] 26(b)(2)(c) apply to all aspects of discovery").
FN11 Fed. R. Civ. P. 26(b)(3); see Lockheed Martin Corp. v. L-3 Comm. Corp., 2007 WL 2209250 (M.D. Fl. July 29, 2007) ("documents containing instructions about how to conduct the [ESI] search and what specifically to search for are opinion work product" and therefore protected as attorney work product privileged material); see also Gibson v. Ford Motor Co., 2007 WL 41954, at *6 (N.D. Ga. Jan. 4, 2007) (document retention notice that included a list of search terms reflected attorney mental impressions and so constituted protected work product).
FN12 Victor Stanley, 250 FRD at 256, United States v. O'Keefe, 537 F.Supp.2d 14 (D. D.C. 2008), and Equity Analytics, LLC v. Lundin, 248 FRD 331 (D. D.C. 2008).
FN13 Counsel, early in the process, should consider disclosure of keywords and other aspects of the search protocol to the adversary and the court, and invite their comment and approval, as a means of managing discovery costs and risk. Magistrate Judge Grimm in Victor Stanley Inc., 250 FRD at 256, found, among other things, that defense counsel's failure to disclose the keywords used to screen for privileged documents in defending its search methodology justified a finding of waiver as to the inadvertently produced documents. The court provided a checklist for attorneys to follow when preparing the methodology to be used to gather and produce ESI: Attorneys should consider the reasonableness of "the keywords used; the rationale for their selection; the qualifications of the [creators of the search] to design an effective and reliable search and information retrieval method; whether the search [is] a simple keyword search, or a more sophisticated one, such as one employing Boolean proximity operators ... ." Id.
FN14 Disability Rights Council v. Washington Metropolitan Transit Authority, 242 FRD 139 (D. D.C. 2007), citing George L. Paul & Jason R. Baron, "Information Inflation: Can the Legal System Adapt?" 13 Rich. J.L. & Tech. 10 (2007).
FN15 TREC 2007 Legal Track.
FN16 Victor Stanley, 250 FRD at 261 n.10.
FN17 http://www.ediscoveryinstitute.org/research/index.html.
FN18 Such tools are described in further detail in The Sedona Best Practices at 191-216 & Appendix. While we are unaware of a court that has expressly endorsed this approach, parties have used these search techniques in conducting, among other things, internal investigations.
FN19 Magistrate Judge Grimm in Victor Stanley, 250 FRD at 260, and Magistrate Judge Facciola in O'Keefe, 537 F.Supp.2d at 24, and Equity Analytics, 248 FRD at 333, have indicated that conducting and defending e-discovery may at times require experts. Indeed, Magistrate Judge Facciola stated that search methodologies in e-discovery may be scrutinized under Rule of Evidence 702.

Labels: Boolean Search, Carrot2, Conceptual Search, Daubert, dtSearch, eDiscovery, Keyword Search, Lucene, Nutch, Rule 702, Transparent Search

posted by Charles Skamser @ 9:42 AM 0 Comments

2009 eDiscovery Predictions and Trends

In the process of completing my research to write my first annual 2009 eDiscovery Predictions and Trends, I came across an excellent guest Blog post titled "2009 Predictions and Trends" on the e-Discovery Team Blog by Sonya Sigler, General Counsel and V.P. of Business Development at Cataphora, Inc.

And although this will not replace my 2009 eDiscovery Predictions and Trends, I thought that it provided some outstanding insight into what we should all expect in 2009. The major topics of her predictions fall into the following categories:

Labels: 2009 eDiscovery Predictions and Trends, Conceptual Search, dtSearch, FRCP, Qualcomm, Ralph Losey, Sedona Conference Cooperation Proclamation, Sonya Sigler, Zubulake

posted by Charles Skamser @ 11:33 AM 1 Comments