Unstructured documents analysis

[wassfila]

Unstructured documentation can include unorganized documents from different sources, and not necessarily following a clear folder structure. Therefore the lack of structure occurs at two levels

• document source, path and metadata
• internal document layout and content

Frameworks libraries tools and services

• https://docs.aws.amazon.com/bedrock/latest/userguide/kb-how-it-works.html : full fledges RAG service (chunking, vectordb, llm) multiple sources s3, confluence, sharepoint, web,...
• https://aws.amazon.com/kendra/ : enterprise grade (latency, complexity, cost) doc analysis, storage and nlp retriaval all in one, even more sources than knowledge base
• https://aws.amazon.com/textract/ : images and rendered PDF as images for ocr and layout extraction, e.g. tables, forms,...
• https://docs.unstructured.io/welcome : ingestion platform for RAG, multiple formats supported pdf, doc, csv, pptx, xlsx, xml, md, png,... dependencies : tesseract, libreoffice, pandoc,...
• https://github.com/VikParuchuri/marker : (gpl+commercial) PDF to markdown convertion using surya for ocr and layout identification
• https://github.com/tesseract-ocr/tesseract ocr library but lacks layout identification

Blogs and references

• https://www.altexsoft.com/blog/unstructured-data/ : classification of unstructured data, comparision, tools listing,...
• https://aws.amazon.com/comprehend/features/ : clasical ml oriented service for entity recognition, sentiment analysis,...

[wassfila]

AI rewrite

Unstructured Documentation: Challenges and Solutions
Unstructured documentation often includes disorganized documents from various sources without a consistent folder structure. This lack of structure appears at multiple levels:

• Source, Path, and Metadata: Documents originate from mixed sources with inconsistent metadata and storage paths.
• Internal Layout and Content: Documents display varied internal structures, which complicates content parsing and data extraction.

Recommended Tools and Services for Managing Unstructured Documentation

1. AWS Bedrock: Comprehensive retrieval and generation (RAG) service suitable for handling data from multiple sources such as S3, Confluence, SharePoint, and web. Supports chunking, vector databases, and language model integrations.

2. Amazon Kendra: An enterprise-grade search service designed for complex environments, featuring document analysis, storage, and NLP-based retrieval. It supports a broader range of data sources compared to other knowledge bases.

3. Amazon Textract: Specializes in OCR and layout extraction from images and rendered PDFs, such as extracting data from tables and forms.

4. Unstructured.io: An ingestion platform that supports multiple file formats (PDF, DOC, CSV, PPTX, XLSX, XML, MD, PNG) and integrates well with RAG systems.

5. Marker by Vik Paruchuri: A tool that converts PDF documents to markdown, using Surya for OCR and layout recognition.

6. Tesseract OCR: A widely-used open-source OCR library. It is excellent for text recognition but does not provide layout identification, which may need to be supplemented with additional tools for comprehensive document parsing.

By focusing on these tools and services, organizations can better manage unstructured documents through automated processing, improved accessibility, and efficient information retrieval.