Skip to main content

Extract text from files

Speed up document ingestion by extracting text from files for processing​

With Squid's text extraction feature, you can easily process the contents of files for fast turnaround of information. From compliance documentation to earnings reports to scientific studies, there are many types of documents that need to be reviewed and actions that must be taken based on their contents.

Manual ingestion of this information is time-consuming and error-prone. With Squid, these tasks that previously took days are completed in seconds.

Note

Text extraction requires admin privileges, so it should only be performed in a secure environment with access to your Squid API key like the Squid backend or other server environment.

Create the extraction client​

To perform text extraction on a document, first create an extraction client using the extraction() method:

Backend code
const extractionClient = this.squid.extraction();

Extract the text​

Use the extraction client's extractDataFromDocumentFile method to extract text from the file. The method takes one of two types: File or BlobAndFileName. The extractDataFromDocumentFile method returns a promise that resolves to an array of pages. The text of a given page can be accessed using the content attribute.

The following example shows extracting text from a file:

Backend code
const data = {
  blob: dataBlob,
  name: 'myDocument.pdf',
};

const extractedResult = await extractionClient.extractDataFromDocumentFile(data);
console.log(extractedResult.pages[0].content); // 'Q4 Development Plan...'
What content is extracted?

Squid's extraction client uses powerful AI handling to extract text from a variety of file contents. Not only does it support reading text files, but it will also extract text from scanned documents, tables, multiple languages, and more.

When the OCR provider reports how many pages it processed, the result also carries that count as ocrPagesProcessed. Use it to track billable pages rather than counting pages, since the two can differ: a blank page is processed and billed but returns no content.

Backend code
// May exceed extractedResult.pages.length when the document contains blank pages.
console.log(extractedResult.ocrPagesProcessed);

The extraction client has an additional method for extracting text from a document at a remote URL. Both extraction methods optionally include an options parameter that allows you to specify the pageIndexes of the document to extract, the extraction method to use, and options for image extraction:

Backend code
const extractedResult = await extractionClient.extractDataFromDocumentUrl(
  'www.file-url.com',
  { pageIndexes: [0, 1, 2] }
);

Extraction methods​

Squid extracts documents through one of several methods. To choose one, pass preferredExtractionMethod in the options parameter; the same option is accepted when uploading file contexts to a knowledge base. As the name suggests, the method is a preference rather than a guarantee, and requesting one never causes an error: the two Basic methods are always honored, but when the provider behind a provider-backed method isn't configured in your environment, Squid silently falls back to the first available provider-backed method (in the order Mistral OCR, Amazon Textract, Azure AI Document Intelligence), and to plain Basic (legacy) when none are available. When you don't choose a method, your environment's default method goes through the same availability fallback. The extraction response does not report which method actually ran.

  • mistral_ocr (Mistral OCR), amazon_textract (Amazon Textract), azure_document_intelligence (Azure AI Document Intelligence): provider-based OCR. These are the most capable extraction methods because they are able to identify figures/graphs on pages even if they are not embedded images, leading to the most accurate extraction of your documents.
  • legacy (Basic): reads the PDF's text layer and embedded images directly. Fast, but a scanned page has no text layer and yields no text. Note that the embedded images it extracts are described by AI when the document is ingested into a knowledge base (see extractImages below).
  • legacy_with_llm (Basic + page understanding): the same text extraction as Basic, plus an image of each page described by AI. This can be necessary as not every figure/graph appears as an embedded image. This costs one AI call per page, so long documents extract more slowly and at higher cost than Basic.

The names in bold are the labels shown in the Squid Console's knowledge base Add Knowledge dialog, where the same choice is available when uploading documents. The method preselected in that dialog is your environment's default.

Backend code
const extractedResult = await extractionClient.extractDataFromDocumentFile(
  data,
  { preferredExtractionMethod: 'legacy_with_llm' }
);

Two companion options control image handling for every method. extractImages defaults to true; images extracted from a document are described by AI when it is ingested into a knowledge base, and setting it to false skips images entirely, including Basic + page understanding's page descriptions. imageMinSizePixels defaults to 100; embedded images smaller than this in either dimension are skipped.

Next steps​

Once your text is extracted, you can:

  • Parse the text and write to a database using Squid's database connectors.
  • Pass text as part of a query to a Squid AI Agent to answer questions or take actions based on the text.