Do Litigators Need to Understand Predictive Coding Theory?

With the debate over Predictive Coding entering a feverish pitch, there is an interesting thread of discussion beginning to emerge asking whether or not litigators and other users need to understand what I am going to refer to as Predictive Coding Theory.

In a May 4, 2012 Blog  titled, "3 Drawbacks To Predictive Coding", Sandra E. Serkes, President and CEO of Valora Technologies writes "What is missing (in regards to the Predictive Coding debate), is a discussion of the specific weaknesses of the overall Predictive Coding technique.   She then goes on to indicate that, "Predictive Coding tagging algorithms are not transparent".

To put this into more technical terms, do we need to know what probability theories and related  dimension reduction systems are being used as the foundational algorithms for Predictive Coding system to identify relevant documents?

For those of you who are interested in a more detailed overview of Predictive Coding, I suggest that you read a March 25, 2012 Blog post titled, "Predictive Coding Based Legal Methods for Search and Review", Ralph Losey does an excellent job of discussing the basic technical mechanics and some of the underlying theories of Predictive Coding.

Getting back to my question about how much we need to know about Predictive Coding, I am in the process of developing some unique insight.  Over the past 30 days, in preparation for adding a Predictive Coding module to the DCIG/eDSG 2012 Early Case Assessment Buyers Guide, I have been interviewing product managers from some of the Predictive Coding vendors and current users of Predictive Coding system to develop a list of criteria for reviewing and ranking the platforms for our buyers guide.  One of the questions that I have been asking is what probability theories and related  dimension reduction systems are being used as the foundational algorithms for your Predictive Coding platform to identify relevant documents.

So far, I haven't gotten a straight answer as most of the product managers either don't undesrtand the question or want to move the discussion up a couple of layers in technology stack to talk about indexing, semantic search, clustering, relevance ranking, sampling and presentation of results.  There is no doubt that these are all very pertinent topics to a perspective buyer of Predictive Coding technology.  However, it doesn't answer the question about the transparency of exactly how these systems are identifying relevant documents.

Whether or not litigators need to understand Predictive Coding theory and the underlying probability theories and related  dimension reduction systems is debatable.  However, I believe that a minimum level of transparency from the Predictive Coding vendors would at least give buyers the opportunity to understand what they are buying and then compare the various offerings.

In 1969, Edgar F. Codd and some of his associates that I have actually had the honor or knowing,  first formulated and proposed the theory of relational database.  And, although I am not sure that it reach the level of skepticism and resistance to blind adoption that we are currently seeing with Predictive Coding.  However, it was new and therefore many did require an explanation of the underlying theories and mathematics.  Eventually,  the discipline normalized and everyone just assumed that relational databases worked and there was no longer any need to question "how they worked'.

A similar vetting process would be very healthy for Predictive Coding.  Check back to my blog in the coming weeks for updates and more information on this topic.

Labels: Clustering, eDiscovery, Litigation, Predictive Coding, Relevance Ranking, Search, Semantic Search

posted by Charles Skamser @ 9:07 AM 2 Comments

The Top 5 Most Requested Features for Early Case Assessment Platforms

The week of April 9, 2012, eDiscovery Solutions Group (eDSG) ran a poll that asked voters to rank the top 5 most important features for an Early Case Assessment (ECA) platform.  Having spent the last 6 months analyzing ECA technology from the top ECA vendors in the world and talking to hundreds of end users in preparation for the release of the 2012 DCIG/eDSG Buyers Guide, the results of this survey were not overly surprising.  However, the results are very telling in regards to the paradigm shift that has occurred in how users view Early Case Assessment.  Therefore, I wanted to publish the details of the results of this poll.

Summary of Results
54% of the voters ranked Advanced Analytics as one of the top 5 most important features for an Early Case Assessment (ECA) platform. Support for Collection of Social Media and Collaboration came in second with 31%. Cost Calculations, Predictive Coding, Delivered as SaaS and Semantic Search came in third with 23%.

Following is an analysis and commentary for each of the categories:

Advanced Analytics
I was not surprised that 54% of the voters ranked Advanced Analytics as one of the top 5 most important features for an Early Case Assessment (ECA) platform. After all, ECA is all about "analyzing" your potential pool of ESI to determine where it is located, what you have and how it will affect your case strategy.  However, I thik that there is a subtle hint hidden within this top ranking that may signal a change in way users view the process of eDiscovery . As a followup, it would be interesting to ask users to define "Advanced Analytics" and then identify where within the EDRM, they believe these analytics should be utilized.

Historically, the standard eDiscovery workflow followed the EDRM and therefore ESI that was potentially responsive or relevant was identified via standard file type, custodian and data parameters, collected or ingested, culled. analyzed and reviewed.  This analysis stage is what many within the industry now refer to as Early Case Assessment (ECA).  However, in 2012, there are technologies that allow for identification and analysis of ESI before it is even ingested.  Further, there is also now growing demand for the identification and analysis of live ESI residing in Content Management Systems (CMS), archiving systems, a wide variety of private and public email systems, public cloud storage and even public social media stores.  As a result of these changing requirements, I contend that the fundamental definition of Early Case Assessment (ECA) is evolving and therefore it may be time to adjust best practices, workflows and our expectations of ECA platforms should support.

Support for Collection of Social Media
I was also not surprised that 31% of the voters ranked Support for Collection of Social Media as one of the top 5 most important features of an Early Case Assessment platform. Corporate legal departments, corporate IT departments and outside counsel are beginning to realize the requirement to harvest potential evidence from social media sources such as Facebook, LinkedIn, Twitter, MySpace and Google+ is real and therefore they want their ECA platform to be able to handle these file types.

Collaboration
31% of the voters ranked Collaboration as one of the top 5 most important features of an Early Case Assessment platform.  As the use of social media and SaaS based communication and collaboration applications has become more mainstream and therefore accessible to the enterprise knowledge worker, their expectations have risen accordingly. Therefore, it is not surprising that users expect their ECA platforms to follow the collaboration trend and support communications and collaboration for multiple different types of users from multiple different geographically regions.  Security is obviously an issue that the ECA vendors will have to address as the information that ECA platforms is more sensitive and can lead to sanctions and financial penalties.

Cost Calculations
23% of the voters ranked Cost Calculations as one of the top 5 most important features of an Early Case Assessment platform.  It has been well established that 70% of the cost of eDiscovery is in the cost of review and therefore there is a renewed emphasis on reducing the amount of ESI that eventually has to be reviewed.  With this requirement in mind, users have  been relying upon gathering the appropriate information such as amount of ESI by file type and custodian, etc. and plugging it into an Excel spreadsheet to forecast the eventual full cost of eDiscovery.  As such, I completely understand that users now want this capability added to the ECA platform.

Predictive Coding
I was very surprised that 23% of the voters ranked Predictive Coding as one fo the top 5 most important features of an Early Case Assessment platform.  Predictive Coding currently normally resides as an alternative option to manual tagging and review within the Review section of the EDRM. So, I am not sure if these voters are confused or are truly contending (with their vote) that Predictive Coding technology needs to be pushed further to the left and utilized during the early stages of ESI identification, collection and analysis.

I predict that Predictive Coding or some form of Machine Driven Analysis will be incorporated into the early stages of ESI identification, collection and analysis drvien by the demands to increase operational efficiencies and reduce costs.  Pleaes note that eDSG in partnerhsip with DCIG will be releasing a Predictive Coding Buyers Guide this summer.  And, without giving away any vendor secrets I can state that the market is going to intriqued with the extent to which Predictive Coding or Machine Driven Analysis is going to be available to support the entire eDiscovery lifecycle.

Delivered as SaaS
23% of the voters ranked SaaS Delivery as one fo the top 5 most important features of an Early Case Assessment platform.  As I have said many times, I believe that the "train has left the station" on SaaS based eDiscovery platforms.  The value proposition of operational efficiencies and costs savings are just too high for this train to turn around and therefore I was a bit surprised that Delivery as SaaS wasn't ranked higher.  I suspect that maybe users are now expecting this feature to a standard option of any ECA platform and therefore failed to vote for it.  Please note that I did not inlcude an option to vote for eDiscovery in the Cloud as that has become a very confusing and soon to be highly debated topic of disucssion.


First Pass Review
I was very surprised that only 15% of the voters ranked First Pass Review as one of the top 5 most important features of an Early Case Assessment platform.  As stated previously, it has been well established that 70% of the cost of eDiscovery is in the cost of review and therefore there is a renewed emphasis on reducing the amount of ESI that eventually has to be reviewed.  Therefore the concept of a First Pass Review (FPR) or a mini review done during ECA has become a well accepted practice before a final review set is defined and exported to a full blown review platform.

I predict that with the advancement of Predictive Coding techniques and other associated technologies that First Pass Review is going to expand and cannibalize a major percentage of the current Document Review platform market.  And, at some point in time, what we now define as Document Review will vanish as ESI will go directly from storage or real-time location to analysis and review and then directly into some digital form ready for the courtroom.  As evidence of this trend, it is interesting to note that we are now seeing many of the legacy ECA vendors expanding their FPR capabilities into full Document Review and many of the legacy Document Review vendors expanding their capabilities into ECA.

Workflow Management
I wasn't overly surprised that only 15% of the voters ranked Workflow Management as one fo the top 5 most important features of an Early Case Assessment platform.  I believe that workflow management is an extremely important feature of any operational application. However, I think that the definition of the workflow within the eDiscovery lifecycle is in such a state of flux that it may be premature to try and define a workflow management system that vendors can be confident will be pertinent for more than just a few release cycles.  That being said, I believe that an ECA platform should provide solid workflow for First Pass Review and Predictive Coding and at least a user dashboard outlining the status of a "project" or "matter" as it makes its way from identification to collection to analysis to review to production.  In my opinion, the real answer is a flexible workflow management system that enables users to define their eDiscovery lifecycle. Whether this is accomplished through an integrated OEM solution or as a proprietary solution will be up to the individual vendors to decide.

Legal Hold
Only 8% of the voters ranked Legal Hold as one of the top 5 most important features of an Early Case Assessment platform because many beleive that it is outside the current definition of ECA and  they still don't know if they want to utilize a best-in-class or a holistic approach to support the entire eDiscovery workflow.

I predict that adding in real-time data mapping and coupling it with real-time legal hold that actually preserves ESI in a forensically sound manner is the direction that the market is going to take.  Whether this is accomplished through an integrated OEM solution or as a proprietary solution will be up to the individual vendors to decide.

Real-time Collections
8% of the voters ranked Real-time Collection as one of the top 5 most important features of an Early Case Assessment platform.  Given where eDiscovery has come from in the past 5 years, Real-time anything is a very aggressive position to take.  However, as the enterprise IT organizations become more involved in the eDiscovery process and eDiscovery becomes a subset of the Information Governance process, Real-time identification, analysis and collection will become an industry accepted best practice.

Summary
This poll was very telling in regards to how users are changing their views and associated requirements for what an Early Case Assessment (ECA) platform should support.  ECA is definitely moving further to the left in the eDiscovery lifecycle and it appears that users want the eDiscovery platform vendors to support this paradigm shift.

Click Here to review the results of all of the polls that eDSG has been running  on a weekly basis since June of 2011 on the current status of Information Governance and eDiscovery.  Click Here to vote in the most recent eDSG Weekly Poll.

Labels: Early Case Assessment, ECA, eDiscovery, FPR, Frist Pass Review, Predictive Coding, Saas, Semantic Search

posted by Charles Skamser @ 8:48 AM 0 Comments

Information Governance and eDiscovery Trends for 2012

2011 has been a transitionary year for information governance and eDiscovery.  Enterprises and governments worldwide have had to come to grips with the sudden acceleration of the use of social media, cloud computing, mobile devices and the resulting explosion in the volume of Electronically Stored Information (ESI).

Some are predicting the end of email in favor ot texting, Twitter and other social media electronic communications.  There are over 100 million users on Twitter each day with 35% utilizing a mobile device.  Facebook now has over 800 million users with 74% outside the United States and 350 million on mobile devices.  There are also over 100 million LinkedIn users worldwide as business social media and the assocication collaboration and communications is now required as a fundamental marketing tool for any serious business player.  Click Here for a more detailed graphic overview of the the growth of social media from the Search Engine Journal.

In addition to social meida, the fundamental infrastructure matured in 2011 to support more serious cloud computing.  Enterprises worldwide began to realize the dramatic cost benefits and potential business benefits of cloud computing.  As a result they gave serious consideration to private cloud implementations and are becoming more comfortable with moving some ESI to public Cloud Service Providers (CSPs) such as Amazon and Rackspace.  Microsoft launched Office360 to enable users to access the Microsoft Office Suite of products as Software-as-a-Service (SaaS) removing the expensive and time consuming software update dance that we all dread.  And, late in 2010, Apple released the iCloud platform for consumer cloud storage and set the stage for literally millions of iPhone, iTouch and Mac users to begin storing their pictures and other personal data in the cloud.

For anyone involved in the world of Information Technology (IT) whether as a user or a technologist, 2011 was definitely an exciting year.  For those of us in the information governance and eDiscovery business, 2011 marked a turning point, both literally and conceptually, for the monitoring, indentification, collection, processing, review and production of electronic evidence.

Is it no longer just an interesting discussion at the local pub to talk about the eventuality of social media as a source of evidence.  There were already close to 1,000 cases in 2011 where the judge issued a written opinion that mentioned social media. Likewise, information governance and eDiscovery in the cloud is longer just an interesting breakout session at tradehows to fill up your 'dance card".  Harvesting ESI from the cloud for the purpose of eDiscovery was a real issue in 2011 and therefore anyone that is serious about eDiscovery from a process, legal and technology standpoint, had better start to pay attention.

Given all of this (and more that I have not even mentioned), 2012 is going to be a banner year for changes in information governance and eDiscovery.  Following are my predictions:

Information Governance and eDiscovery Will Move to the Cloud
As more and more ESI moves to the cloud, the requirement to harvest this ESI "from the cloud" will also continue to grow.  However, there will be a tremendous amount of confusion in regards to exactly what information governance and eDiscovery in the cloud means.  Most, if not all, of the seroius technology vendors will announce that they are cloud ready or work in the cloud.  However, in most cases, this will mean that they are using the same old legacy technology and harvesting data "from the cloud" and processing it the same old way they have been processing ESI and paper for years.  Some of these vendors will have their own data centers and offer up information governance and eDiscovery as as Service.   However, this is nothing new.  The real change that we will witness in 2012 will be virtual technology (not hardware based) that enables users to move information governance and eDiscovery solutions/platforms to the cloud ESI and collects it and processes it in the cloud where it resides.  Physical location will no longer be an issue.  Collection and moving massive amounts of data will no longer be an issue.  The salability of hardward solutions will no longer be an issue.  Staging information governance and eDiscovery respositories in third party service provider data centers will no longer be required.

Enterprises will be able to move virtual information governance and eDiscovery solutions around their networks and private clouds as required to collect and process ESI where it resides. Cloud Service Providers (CSPs) will provide information governance and eDiscovery solutions as part of their standard IaaS and PaaS technology stacks.   Early Case Assessment (ECA) is going to take on a whole new meaning.  This approach is a major paradigm shift in the entire concept of how information governance and eDiscovery should work.  2012 is going to be an exciting year for information governance and eDiscovery in the cloud.

Semantic Search will Go Mainstream
The mystical world of semantic search and natural language processing technology will finally go mainstream in 2012.  New Information governance and eDiscovery technology vendors will emerge that are utilizing this technology as the foundation of their offerings or have seamlessly integrated this technology into their platforms.  Litigators will begin to understand the value of semantic search, courts will begin to accept the results and users will begin to demand its use.  Before the end of 2012, the industry may even recognize the term "Lucene".

Predictive Coding will Go Mainstream
Just like the mystical world of semantic search and natural language processing technology, Predictive Coding will also go mainstream in 2012.  New Information governance and eDiscovery technology vendors will emerge that have seamless integrated this technology into their platforms.  Litigators will begin to understand the value of Predictive Coding, courts will begin to accept the results and users will begin to demand its use.  Offshore coding will become less attractive as Predictive Coding will enable all but the very detailed review and tagging to be done automatically.

Informationn Governance and eDiscovery for Social Media Will be Required
Pushed by the ethical requirements for litigators to understand and utilize eDiscovery for social media and the practical requirements to ensure that all pertinent ESI is being collected and submitted, there will be a  major move by the courts and litigators in 2012 to ensure that social media evidence is being collected and submitted with an appropriate chain of custody and with access to metadata to ensure the validity of the evidence.  And, as a results, legacy ECA vendors will social media file types to their bag of tricks and numerous new Discovery tools for social media will be released in 2012.

End-to-End Information Governance and eDiscovery Solutions
As Early Case Assessemnt (ECA) vendors expand their product lines "right" into document review and case management and as document review platform expand their product lines "left" into ECA, the market is going to have a choice of more end-to-end information governance and eDiscovery solutions.  In addition, we will see the integration of data mapping, legal hold, semantic search, predictive coding, project management, workflow management and case management into these end-to-end solutions.

Litigator Will Become More Technology Savy
Out of both practical necessity and statute ordered requirements, litigators will become more technology savvy in 2012.  The days of the hands-off, don't bother me with the technical details for eDiscovery litigator are quickly coming to a close. 

Information Governance and eDiscovery Pricing will Drop
The move to the cloud along with easily scalable virtual solutions offered up as Software-as-a-Service (SaaS) will enable a dramatic price decrease for information governance and eDiscovery in 2012.

eDiscovery will become a subset of Information Governance
As I have predicted in previous years, as eDiscovery moves to the enterprise, eDiscovery will become a subset of Information Governance in 2012.


In conclusion, 2012 is going to be a very exciting year in information governance and eDiscovery.  Enjoy the rest of 2011 and strap yourself in for a wild ride in 2012.



 

Labels: 2012 Trends, Early Case Assessment, ECA, eDiscovery, FRCP, Legal, Legal Hold, Litigation, Predictive Coding, Project Management, Semantic Search, Worlflow Management

posted by Charles Skamser @ 9:02 AM 0 Comments

The Conceptual Search Game is Finally On!!

I have been writing (probably preaching to some) about advanced search technology, the differences between conceptual search and keyword search and the importance of advanced search technology in both the Early Case Assessment (ECA) and document review phases of eDiscovery for the past 3 years.

Concept Search Cash Law Emerging http://ediscoveryconsulting.blogspot.com/2008/06/concept-search-case-law-emerging.html
Concept Search vs. Keyword Search http://ediscoveryconsulting.blogspot.com/2008/12/concept-search-vs-keyword-search-in.html
Litigators Need ESI Analytics – Not Boolean Search Tools http://ediscoveryconsulting.blogspot.com/2010/05/litigators-need-esi-analytics-not.html

However, I have been somewhat disappointed in regards to the level of adoption of true conceptual search technology by the leading Litigation Technology vendors.  That appears to be changing.  As an example, in a June 17, 2010 post by StoredIQ titled, “Email Search: Nowhere to Hide”, the author provides an overview of StoredIQ’s search technology with specific focus on Natural Language Processing (NLP).  The reason that I am pointing this out is that 12 months ago, StoredIQ would not have been spending marketing dollars or Blog space on Natural Language Processing (NLP) because the market didn’t know what it was and didn’t care.

In this Blog post, StoredIQ now contends, “Probably of greatest interest to litigators during the discovery process is StoredIQ’s ability to perform natural language processing (NLP), which is the ability to extract linguistically derived natural language concepts from within email and user files including people, places and things. Legal teams can immediately search using over 250 out-of-the-box concepts and attributes including credit card accounts, social security numbers and stock symbols. NLP identifies word usage based upon context within a sentence. For example, NLP can identify if the word ‘will’ is used to identify a person’s name, a legal document or an auxiliary verb showing intent. StoredIQ has proprietary technology for adaptive sentence boundary disambiguation (ASBD) which substantially increases the precision of Natural Language Processing to address common grammatical deficiencies that are present in many business documents. No other information management technologies have this capability. NLP is a critical capability necessary to accurately perform eDiscovery, records management or risk management as full text indexes alone cannot provide the required level of precision.

Interestingly enough,  in my discussions with General Counsel and their litigation support teams from the Information Technology departments over the past 6 months, I have found a new awareness and appreciation for true conceptual search or semantic search or NLP.    So, StoredIQ is on the right track with their current product  offerings and I would bet that they have a product roadmap with more of the same.

So, I guess the conceptual search game is on and the other litigation technology vendors had better take notice of what their clients are saying in regards to what search technology they need.

The full text of the  StoredIQ Blog post is as follows:

A recent article by Jacob Goldstein, 23 Things Not To Write In An Email, illustrates the type of granularity as well as breadth of keywords that can be used by litigators during the legal discovery process to search for relevant information. He points out some keywords that may raise a legal red flag and should be used carefully when constructing emails. However, today’s technology search capabilities provide such precise, complete and accurate results, that there just isn’t anywhere to hide.
For instance, StoredIQ’s advanced search capabilities can look within compressed files, email archives and email attachments, in addition to the text contained in the email message itself. It can also search non-printable text within a document or email and can search through comments and revisions. In addition to search using keywords, StoredIQ supports many advanced search capabilities including:

• Single term search
• Multiple term search
• Concept-based search
• Boolean operators
• Logical grouping of terms
• Wildcards within search terms or Boolean expressions
• Proximity searches
• Natural language entities
• Regular expressions
• Macro-based searches
• Object level attributes
• By hash value (digital signatures)

Probably of greatest interest to litigators during the discovery process is StoredIQ’s ability to perform natural language processing (NLP), which is the ability to extract linguistically derived natural language concepts from within email and user files including people, places and things. Legal teams can immediately search using over 250 out-of-the-box concepts and attributes including credit card accounts, social security numbers and stock symbols. NLP identifies word usage based upon context within a sentence. For example, NLP can identify if the word ‘will’ is used to identify a person’s name, a legal document or an auxiliary verb showing intent. StoredIQ has proprietary technology for adaptive sentence boundary disambiguation (ASBD) which substantially increases the precision of Natural Language Processing to address common grammatical deficiencies that are present in many business documents. No other information management technologies have this capability. NLP is a critical capability necessary to accurately perform eDiscovery, records management or risk management as full text indexes alone cannot provide the required level of precision.
I know a lot of these terms can be a mouthful, but the underlying take away is that legal teams have the technology to precisely and accurately search electronic data, including email, making it much easier for litigators to discover data that was at one time hidden from them.

Labels: Conceptual Search, Early Case Assessment, ECA, eDiscovery, email, Keyword Search, Natural Language Processing, Seach Technology, Search Terms, Semantic Search, StoredIQ

posted by Charles Skamser @ 11:59 AM 0 Comments

Computer Aided Review Has Arrived

Labels: Computer Aided Review, Hal 900, Semantic Search, Web 3.0

posted by Charles Skamser @ 6:28 AM 0 Comments

Web 3.0 in eDiscovery

Over the past year I have been following discussions about Web 3.0, the next generation of the Internet. And, even though my wife is yelling at me to get ready to go Christmas shopping, I wanted to get this Blog posting done today as I believe that Web 3.0 is going to have a profound effect on eDiscovery in 2010 and beyond. Not only in terms of what we have to "Discover" but also in terms of how we "Discover". And, I am going to use Christmas shopping in my example of how to use Web 3.0. I am not sure what that says about my priorities. However, here goes.

From a SciFi stand point, Web 3.0 moves us closer to a more "Hal 900" like computer experience. So, instead of asking your computer to help you "find information" by typing keywords into Google, you will ask your computer to help you "complete a more conceptual task" such as as last minute Christmas shopping for my family that includes a stop for lunch with burgers and very large HDTV to watch the Packers play the Steelers. To put this into perspective, experts in the world of search would describe this approach as maturing from keyword search to conceptual semantic based search. And, the process of storing data on the Web in a form that machines can naturally understand, or converting it to that form, is called the "Semantic Web", a phrase attributed to Tim Berners-Lee, inventor of the Web (it wasn't Al Gore) and Director of the World Wide Web Consortium (W3C).

Web 1.0 was basically just a static library of information that user could access and utilize much like checking a book out of a library. Web 2.0 (the current state of the Internet), enables users to actually change the content of pages with additional information effectively making 'Web pages" living pages that grow through interaction with visitors. Web 2.0 is also characterized by social networking such as facebook and Linkedin, mass sharing of information via application such as Twitter and access via devices beyond the traditional computer.

Berners-Lee disputes the existence of Web 2.0, calling it nothing more than meaningless jargon [source: Register]. Berners-Lee maintains that he intended the World Wide Web to do all the things that Web 2.0 is supposed to do. His vision of the future Web is similar to the concept of Web 3.0. Unfortunately, the Web's current structure of storing information has been designed to work best with the the keyword approach. And therefore, accept is a few isolated instances, Web 3.0 isn't really even a possibility in 2009. For the realities of Hal9000 interactive with the Web, there have to be some changes made to way information is stored and processed. Web 3.0 will have to have collections of information called ontologies, a file that defines the relationships among a group of terms.

Back to our Christmas shopping task, in order for Web 3.0 to be able to actually provide the information that I need to have a successful shopping trip, eat lunch and watch the game, Web 3.0 will have to have an ontology that understand my family, what they want for Christmas, the best places to get those gifts, restaurants that serve hamburgers that will also be Packer game, etc., etc. etc. Basically, much more information than is available today and more information that will be available anytime soon on the Web. However, this doesn't mean that the technology is not available. Semantic search research and development has been underway for many years and is actually already being used successfully in eDiscovery and other information management markets . It's use is just not very widespread and there is still a tremendous amount of confusion about how it works and the validity of the results (please note that I have written about conceptual/semantic search in eDiscovery on this blog several times).

So, what will eDiscovery look like in the world of Web 3.0. First of all, it won't be a digital transition (i.e. we won't wake up one morning with Web 3.0). It will be a more gradual analog transition with application providing the best ROI emerging first and the rest of the world following behind (i.e. classic early adaptors vs. laggers). And, as already indicated, eDiscovery is already the perfect arena for semantic search as users can isolate information, develop ontologies and then ask the Hal900 type questions like, "what relationships exist between these custodians that are not apparent?". Or, "tell me about potentially important relationships that may exist outside of the known custodian?". Eventually, as the ontologies and the yet unknown upstream next generation ontology type technologies emerge, new search technologies will be able to find just about any information that lawyers require to ensure they understand everything that they need to know to proceed with their matters with a very high level of confidence. And, more importantly, a much higher level of confidence that what could ever be provided by completing the same analytics manually.

Other markets that are also already benefiting from this technology are consumer marketing (uncovering buying patterns), anti-crime and anti-terrorism and drug research (find unknown drug interations), just to name a few.

What is even of more interest to me is how our laws will evolve in this new world under Web 3.0. Just as Hal900 will enable law enforcement and the legal system to better understand and analyze information, so to will this same technology enable criminal elements to become more sophisticated in their approaches. I could very easily envision scenarios where Hal900 is asked the question about the most efficient and least risky way to commit a crime.

It should all be very ineresting and exciting to watch.

Labels: Conceptual Search, eDiscovery 3.0, Ontology, Semantic Search, Web 3.0

posted by Charles Skamser @ 10:24 AM 1 Comments