Skip to main content

Knowledge Bases

Store searchable context for your AI agents, and manage it with metadata schemas, automatic extraction, and query-time filtering.​

Why Use a Knowledge Base​

A knowledge base stores the context your AI agents pull from when answering questions, and is the same as the Knowledge Base ability in the Agent Studio. Adding context allows an agent to provide relevant answers on specific topics that may not be part of the underlying AI model.

The following are simple code examples, though the context you add can be much more complex. Some good examples of context include resources like code documentation, product manuals, business operations (e.g., store hours) and user-specific data. You can mix and match context types to create a robust knowledge base for your AI agent, ensuring that it can provide any information your users will need.

Creating a Knowledge Base​

To add or update agent context, you must first create and connect a knowledge base.

First, create the new knowledge base with an embedding model that we provide out of the box:

Client code
await squid.ai().knowledgeBase('banking-knowledgebase').upsertKnowledgeBase({
  description: 'This Knowledge Base contains information on card data',
  embeddingModel: 'text-embedding-3-small',
  chatModel: 'gpt-5.5',
  metadataFields: [],
});

Or, you can use an integration-based embedding model by passing an object with the connector ID, model name, and dimensions. See the OpenAI Compatible Embedding connector for setup instructions.

Client code
await squid
  .ai()
  .knowledgeBase('banking-knowledgebase')
  .upsertKnowledgeBase({
    description: 'This Knowledge Base contains information on card data',
    embeddingModel: {
      integrationId: 'my-embeddings',
      model: 'text-embedding-3-small',
      dimensions: 1536,
    },
    chatModel: 'gpt-5.5',
    metadataFields: [],
  });

A knowledge base stores its vectors in one of two backends: 'mongoAtlas', which supports native hybrid fusion, ranked keyword search (see Keyword search), and knowledge graph search, or 'postgres', which supports none of these. Omitting vectorDbType takes the server default, which depends on the deployment your application runs on. Pass it explicitly when your application depends on a specific backend. The backend cannot be changed after creation, and you can read it back with getKnowledgeBase():

Client code
await squid.ai().knowledgeBase('banking-knowledgebase-atlas').upsertKnowledgeBase({
  description: 'This Knowledge Base contains information on card data',
  embeddingModel: 'text-embedding-3-small',
  chatModel: 'gpt-5.5',
  metadataFields: [],
  vectorDbType: 'mongoAtlas',
});

const kb = await squid.ai().knowledgeBase('banking-knowledgebase-atlas').getKnowledgeBase();
console.log(kb?.vectorDbType); // 'mongoAtlas'

getKnowledgeBase() resolves to the stored knowledge base record (including vectorDbType), or to undefined when no knowledge base exists with that ID. Upserting a different vectorDbType onto an existing knowledge base throws an error rather than changing the backend.

Because the backend is fixed at creation, it also decides whether knowledge graph search can ever be enabled: a knowledge base created on 'postgres' cannot gain a graph. Pass vectorDbType: 'mongoAtlas' rather than relying on the server default if you may want one later.

Upserting Context​

To add context to the knowledge base, use the upsertContext() method, passing the context and its type.

The upsertContext() method accepts a context ID. Providing a context ID allows you to more easily access context later for when you want to make changes.

Client code
const data = `Platinum Mastercard® Fair Credit, No annual fee. Flexible due dates...`;

await squid.ai().knowledgeBase('banking-knowledgebase').upsertContext({
  type: 'text',
  title: 'Credit Card Info',
  text: data,
  contextId: 'credit-cards',
});

Alternatively, use upsertContexts() to upsert an array of contexts.

Client code
const creditCard1 = `Platinum Mastercard® Fair Credit, No annual fee. Flexible due dates...`;
const creditCard2 = `Gold Mastercard®, $50 annual fee. Due dates once a month...`;

await squid
  .ai()
  .knowledgeBase('banking-knowledgebase')
  .upsertContexts([
    {
      type: 'text',
      title: 'Credit Card 1 Info',
      text: creditCard1,
      contextId: 'credit-cards1',
    },
    {
      type: 'text',
      title: 'Credit Card 2 Info',
      text: creditCard2,
      contextId: 'credit-cards2',
    },
  ]);

Connecting to an Agent​

A knowledge base only affects an agent's answers once it is connected to that agent. To connect a knowledge base and configure how the agent uses it, see Connect a Knowledge Base to an Agent.

Context Types​

Two types of contexts are supported: text and file.

Text context is created with a string that contains the context:

Client code
const data = `Platinum Mastercard® Fair Credit, No annual fee. Flexible due dates...`;

await squid.ai().knowledgeBase('banking-knowledgebase').upsertContext({
  type: 'text',
  title: 'Credit Card Info',
  text: data,
  contextId: 'credit-cards',
});

File context is created by providing a File object as a second parameter to the upsertContext() method. The file is then uploaded to Squid and the context is created from the file contents.

Client code
const file = new File([contextBlob], 'CreditCardList.pdf', { type: 'application/pdf' });

await squid.ai().knowledgeBase('banking-knowledgebase').upsertContext(
  {
    type: 'file',
    contextId: 'credit-cards',
  },
  file
);
Note

Your context can be as long as you like; however since there are character limits to LLM prompts, only a portion of your context may actually be included alongside the user's inquiry. When constructing a prompt, Squid decides which portions of the supplied context are most relevant to the user's question.

Spreadsheet Files​

Spreadsheet files (.csv, .tsv, .xlsx, .xlsm, .xls, and .xlsb) uploaded as file context are handled by a dedicated ingestion pipeline. Instead of chunking raw cell text, Squid extracts the workbook's structure (sheet names and sizes, header rows, hidden sheets, and, where the file format provides them, charts and pivot tables) and embeds a generated summary of the whole workbook. Search results for spreadsheet contexts therefore describe what a workbook contains rather than returning fragments of cell data.

Summaries are previews, so agents do not rely on them for exact numbers. When a connected knowledge base contains spreadsheet contexts, the agent automatically gets a querySpreadsheetsWithAi tool that answers questions against the actual uploaded files by running Python in a sandbox:

  • Exact values: counts, sums, averages, lookups of specific rows or cells, filtering, and sorting.
  • Questions spanning several workbooks, such as joining or comparing data across files, in a single call.
  • Structure and provenance questions: what a sheet actually contains, which sheets feed live calculations, and which cells are formulas versus hardcoded inputs. Formula and dependency inspection is fullest for .xlsx/.xlsm, partial for .xls, and unavailable for .xlsb (cell values only) and CSV/TSV (no formula metadata).

No configuration is required, but the dedicated pipeline and the agent tool depend on the retained original file. When a context is uploaded with discardOriginalFile: true (an upsertContext() file option, false by default, that tells Squid to drop the stored original after text extraction instead of keeping it for reprocessing and download), the spreadsheet is ingested as plain extracted text and querySpreadsheetsWithAi is not offered for it.

Getting Context​

To get a list of all contexts, use the listContexts() method. This method returns an array of agent context objects, which includes the contextId:

Client code
await squid.ai().knowledgeBase('banking-knowledgebase').listContexts();

To get a specific context item, use the getContext() method, passing the context ID:

Client code
await squid.ai().knowledgeBase('banking-knowledgebase').getContext('credit-cards');

Listing a page of context​

listContexts() returns every context in the knowledge base, which becomes unwieldy once a knowledge base holds thousands of entries. Use listContextsPage() to page through them instead, and to search by ID or title:

Client code
const page = await squid
  .ai()
  .knowledgeBase('banking-knowledgebase')
  .listContextsPage({
    offset: 0,
    limit: 50,
    // Truncates each entry's text so a listing stays small.
    truncateTextAfter: 500,
    // Case-insensitive substring match on context ID and title only, not on the text.
    search: 'credit',
  });

Every option is optional. The response carries the requested contexts plus a totalCount that ignores offset and limit, so you can render a page count without a second call.

search matches the context ID and title only, so use searching to find entries by their content. truncateTextAfter is available in the TypeScript client only.

Deleting Context​

To delete a context entry, use the deleteContext() method:

Client code
await squid.ai().knowledgeBase('banking-knowledgebase').deleteContext('credit-cards');

This method results in an error if an entry has not yet been created for the context ID provided.

Context Metadata​

When adding or updating the context of an AI knowledge base, you can optionally provide context metadata. Metadata is an object where keys can have a type of string, number, or boolean. Adding metadata provides additional information about the context that can then be used when interacting with your agent. The following example shows adding a PDF as context and providing two key/value pairs as metadata:

Client code
const file = new File([contextBlob], 'CreditCardList.pdf', { type: 'application/pdf' });

await squid
  .ai()
  .knowledgeBase('banking-knowledgebase')
  .upsertContext(
    {
      contextId: 'credit-cards',
      type: 'file',
      metadata: { company: 'Bank of America', year: 2023 },
    },
    file
  );

You can then use metadata when chatting with your AI agent, as shown in the filtering context with metadata section.

Defining a metadata schema​

Declaring a metadata schema on the knowledge base makes metadata structured and self-maintaining. Pass field definitions in metadataFields when upserting the knowledge base:

Client code
await squid
  .ai()
  .knowledgeBase('banking-knowledgebase')
  .upsertKnowledgeBase({
    description: 'This Knowledge Base contains information on card data',
    embeddingModel: 'text-embedding-3-small',
    chatModel: 'gpt-5.5',
    metadataFields: [
      { name: 'author', dataType: 'string', required: true, description: 'The full name of the person who wrote the document.' },
      { name: 'publishedAt', dataType: 'date', required: false, description: 'The date the document was published.' },
      { name: 'category', dataType: 'string', required: false, description: 'The document category, for example report, memo, or guide.' },
    ],
  });

Each field definition has:

FieldTypeRequiredDescription
namestringYesLetters, numbers, and underscores only. Reserved names such as text and contextId are rejected.
dataType'string' | 'number' | 'boolean' | 'date'YesUsed for validation and filtering. date values are normalized to epoch milliseconds so range filters work.
requiredbooleanYesA hint surfaced to the agent alongside the field; it does not affect extraction or ingestion — a context still ingests even when a required value cannot be found.
descriptionstringNoUsed to extract the value from documents and to guide agents when they construct metadata filters.

The TypeScript type also declares an 'array' data type; it is not yet supported and is rejected when the knowledge base schema is saved.

Declaring a schema enables three things: validation of supplied metadata (a value with the wrong type rejects only that context), automatic extraction of missing values, and agent-driven filtering.

Automatic metadata extraction​

When a context is upserted without values for declared fields, Squid fills them in automatically, whether or not the fields are marked required. For file uploads, document properties are used first: fields named after well-known properties (title/docTitle, author/docAuthor, createdAt/docCreatedAt, modifiedAt/docModifiedAt, and docType) are filled from PDF and Office document properties, markdown front matter, and HTML metadata. An LLM pass over the document text handles all remaining fields, guided by each field's description. Values you supply always win and are never overwritten, with one exception: an empty string counts as no value and stays eligible for extraction. The names of extracted fields are recorded on the stored context in autoExtractedMetadataFields, and extraction is best-effort: a value that cannot be found stays absent without failing the upload.

To have Squid write field descriptions for you based on the values already stored in the knowledge base, call generateMetadataFieldDescriptions(). It returns the generated descriptions without saving them; review and save them onto the knowledge base with upsertKnowledgeBase():

Client code
const { fields } = await squid.ai().knowledgeBase('banking-knowledgebase').generateMetadataFieldDescriptions({ overwriteExisting: false });

Filtering Knowledge Base Context with Metadata​

When you have added metadata to your context, you can use the contextMetadataFilterForKnowledgeBase chat option to instruct the AI agent to only consult specific contexts. Only contexts that meet the filter requirement will be used to respond to the client prompt.

The following example filters contexts to only include those with a metadata value of "company" that is equal to "Bank of America":

Client code
await squid
  .ai()
  .agent('banking-copilot')
  .ask('Which Bank of America credit card is best for students?', {
    contextMetadataFilterForKnowledgeBase: {
      ['banking-knowledgebase']: { company: { $eq: 'Bank of America' } },
    },
  });

The following metadata filters are supported:

FilterDescriptionSupported types
$eqMatches vectors with metadata values that are equal to a specified valuenumber, string, boolean
$neMatches vectors with metadata values that are not equal to a specified valuenumber, string, boolean
$gtMatches vectors with metadata values that are greater than a specified valuenumber
$gteMatches vectors with metadata values that are greater than or equal to a specified valuenumber
$ltMatches vectors with metadata values that are less than a specified valuenumber
$lteMatches vectors with metadata values that are less than or equal to a specified valuenumber
$inMatches vectors with metadata values that are in a specified arraystring, number
$ninMatches vectors with metadata values that are not in a specified arraystring, number
$existsMatches vectors with the specified metadata fieldboolean

Scoping to a folder with $underPath​

$underPath matches a hierarchical string field, such as a document's folderPath, against a subtree. It matches when the value is exactly the operand, or begins with the operand followed by a /:

Client code
await squid
  .ai()
  .agent('banking-copilot')
  .ask('What changed in the 2023 reports?', {
    contextMetadataFilterForKnowledgeBase: {
      ['banking-knowledgebase']: { folderPath: { $underPath: 'reports/2023' } },
    },
  });

That filter admits reports/2023 and reports/2023/q1/deck, but never the sibling reports/2023 drafts. The / separator supplies the segment boundary, which a raw prefix match would not, so subtree scoping cannot leak into a folder that merely shares a prefix.

Details worth knowing:

  • Leading and trailing slashes in the operand are ignored, so reports/2023/ and reports/2023 address the same subtree.
  • An empty operand matches every context that has the field at all.
  • Matching is case-sensitive, so Reports and reports are distinct folders.
  • folderPath is an ordinary metadata key rather than a reserved one. Write it with POSIX / separators and no leading or trailing slash, and use the empty string (not an absent value) for a file at the upload root.
  • $underPath applies to knowledge bases only. It is not accepted by metrics tag filters or matchmaking, which reject unknown operators.

A bare scalar value is shorthand for $eq, so { company: 'Bank of America' } and { company: { $eq: 'Bank of America' } } are equivalent. Filters can also be combined with $and and $or:

Client code
await squid
  .ai()
  .agent('banking-copilot')
  .ask('Summarize recent card reports', {
    contextMetadataFilterForKnowledgeBase: {
      ['banking-knowledgebase']: {
        $and: [{ category: 'report' }, { publishedAt: { $gt: Date.parse('2026-01-01') } }],
      },
    },
  });

Fields declared with dataType: 'date' are stored as epoch milliseconds, so pass numeric values (for example Date.parse('2026-01-01')) in range filters.

Agent-driven metadata filtering​

When a knowledge base declares a metadata schema, connected agents can construct metadata filters on their own. The knowledge base search tool gains a filter parameter, and the model fills it in based on the user's question, guided by each field's description and a small set of sample values shown in the tool description.

Filters you set with contextMetadataFilterForKnowledgeBase always apply and are combined with the agent's filter using AND. The agent can narrow the allowed scope but can never widen it, so the app-level filter remains a security boundary.

Setting enableMetadataInspection: true on the connected knowledge base additionally gives the agent an inspection tool that enumerates and searches a field's stored values on demand, which helps it construct accurate filters. Values are sampled from the most recently updated documents, so the absence of a value in the sample does not prove it never occurs.

Respecting Source Permissions​

Content indexed from a connector usually carries its own access rules. A SharePoint document, a Confluence page, or a Slack conversation is visible to some people in your organization and not to others. A knowledge base can honor those rules, so an agent grounds its answers only in the content the person chatting with it is allowed to see.

Two people who ask the same agent the same question receive answers drawn only from the documents each of them can open in the source system. Content a user cannot access is never retrieved, never reaches the model, and never appears in citations.

This is separate from metadata filtering, which narrows results by topic or attribute rather than by who is asking. The two combine, and permissions always apply.

note

Your application remains responsible for authenticating users. Squid applies the permissions of the authenticated user identity your application establishes. See Authentication to set this up.

Searching a Knowledge Base​

Use the search() method to query a knowledge base directly and get back the matching chunks:

Client code
const chunks = await squid.ai().knowledgeBase('banking-knowledgebase').search({
  prompt: 'Which credit cards have no annual fee?',
});

Semantic (vector) search is weakest exactly where precision matters most: exact tokens such as identifiers, error codes, SKUs, and file names. The optional searchMode option selects how matches are found. How each mode behaves depends on the knowledge base's search backend: the vectorDbType ('mongoAtlas' or 'postgres') fixed when the knowledge base is created and readable via getKnowledgeBase().

ModeDescription
'hybrid'The default on a knowledge base without a graph. On 'mongoAtlas', natively fuses semantic and keyword candidates; on 'postgres', falls back to semantic search.
'vector'Semantic similarity only.
'keyword'Embedding-free lexical search. On 'mongoAtlas', ranked full-text (BM25) matching, where a partial match still returns the best results. On 'postgres', an unranked filter: every whitespace-separated term must appear in a chunk as a literal, case-insensitive substring.
'graph'Multi-hop retrieval over an entity graph, fused with a hybrid search. Available only on 'mongoAtlas' knowledge bases with the graph enabled, where it is also the default. See Knowledge Graph Search.
Client code
const chunks = await squid.ai().knowledgeBase('banking-knowledgebase').search({
  prompt: 'ERR_0000_4F2A',
  searchMode: 'keyword',
});

Connected agents choose a search mode on their own: the knowledge base search tool offers the modes the backend supports, and the agent switches to keyword search when a question targets exact tokens. Knowledge bases with a knowledge graph additionally offer a 'graph' mode, which is their default when searchMode is omitted.

Literal scan with grep​

Every search mode above works on chunks, which are produced after ingestion has split and processed the text. grep() scans the raw extracted text instead, before chunking, and returns each matching line with the file it came from. Reach for it when you need characters rather than meaning, and when the surrounding line matters, such as finding a specific SKU in a price list or a value in a spreadsheet row.

Client code
const result = await squid
  .ai()
  .knowledgeBase('banking-knowledgebase')
  .grep('ERR_0000_4F2A', {
    // Scopes the scan to matching contexts before any text is read.
    metadataFilter: { category: 'runbooks' },
    maxMatches: 100,
  });

for (const match of result.matches) {
  // e.g. "CreditCardList.pdf, page 4, line 12: ..."
  console.log(`${match.fileName}, ${match.part}, line ${match.lineNumber}: ${match.line}`);
}

Both options are optional:

OptionTypeDescription
metadataFilterobjectScopes the scan to the contexts whose metadata matches, using the same grammar as metadata filtering, including $and, $or, and $underPath. Applied before any text is matched.
maxMatchesnumberThe maximum number of matches to return. Defaults to 50 and is capped at 200.

Each match carries the contextId and fileName it came from, the part (a sheet or section title when the extractor supplies one, otherwise page N), the 1-based lineNumber, the matching line, and context, which is the matching line plus a small window of surrounding lines.

Two behaviors to keep in mind:

  • The pattern is always literal. Regular expression metacharacters are escaped, so punctuation and symbols match themselves. The pattern may span lines.
  • Matching is case-insensitive for ASCII letters only. acme finds ACME, but café does not find CAFÉ. Search the exact casing when the text is not ASCII.

An empty matches array proves the string is absent only when the response carries no limitation. A scan that fell short of the requested scope sets limitation to one of the following, so check it before concluding anything from an empty result:

LimitationMeaning
timedOutThe scan hit its server-side deadline. Routine on large knowledge bases, and says nothing about whether the pattern occurs.
noIndexedTextNothing in scope has stored text. Contexts ingested before literal search existed stay here until re-ingested.
filterMatchedNoContextsmetadataFilter admitted no contexts, so nothing was read.
scopeTruncatedmetadataFilter admitted more contexts than one scan can cover, so only some were searched. Narrow the filter.
partsCapReachedThe scan filled its per-call budget of matching pages or sheets, so more matching places exist.
partialCoverageSome contexts in scope have no stored text. unscannedContextCount reports how many.
truncatedContentSome in-scope text was cut at ingest and was never searchable. truncatedContentFileNames samples the affected files.
coverageUnknownThe scan succeeded, but how much of the scope holds stored text could not be determined.

The separate truncated flag means only that the returned list stopped at maxMatches and that more matches exist.

Agents connected to a knowledge base get this scan as a grepKnowledgeBase tool, and they broadcast status updates as they run it: grepKnowledgeBase reports the title Searching Knowledge Base Text, distinct from the Accessing Knowledge Base title of ranked retrieval.

grep() is available in the TypeScript client. There is no Python or REST equivalent.

Bulk Ingestion​

upsertContexts() ingests inline and resolves once the contexts are searchable, which suits tens of documents. For thousands, use bulk ingestion: a durable, asynchronous lane that runs contexts through the AI provider's batch APIs.

The difference that matters is the return contract. bulkUpsertContexts() resolves as soon as the request is staged, never when ingestion finishes, so you track the returned job separately.

Ingesting a directory with the CLI​

The quickest path is the CLI, which walks a directory, batches the files, uploads them, and polls each job to completion:

squid kb-upload --dir ./docs --knowledgeBase banking-knowledgebase
OptionDescription
--dirRequired. The local directory to walk recursively.
--knowledgeBaseRequired. The knowledge base to ingest into.
--extensionsComma-separated allow-list. Defaults to pdf, docx, txt, md, html, csv, xlsx, xls, xlsm, xlsb, pptx.
--batchSizeFiles staged per job. Defaults to 200, maximum 1000.
--dryRunLists the files that would be uploaded and exits without contacting the server.
--timeoutMinutesHow long to wait for each job before reporting it as still running server-side and moving on. Defaults to 120.

--appId, --apiKey, --region, and --environmentId fall back to SQUID_APP_ID, SQUID_API_KEY, SQUID_REGION, and SQUID_ENVIRONMENT_ID. Pressing Ctrl-C cancels the job that is in flight.

Staging contexts from code​

Bulk ingestion requires an API key, so run it from backend code:

Backend code
const { jobId, contextIds, duplicates } = await this.squid
  .ai()
  .knowledgeBase('banking-knowledgebase')
  .bulkUpsertContexts(contexts, files);

contextIds is index-aligned with the contexts you passed. A context rejected as a content duplicate still occupies its slot, so cross-reference duplicates by context ID to see what was actually staged. A duplicate is content the knowledge base already holds, or content an earlier context in the same call introduced.

Tracking a job​

Client code
const kb = squid.ai().knowledgeBase('banking-knowledgebase');

// Poll for the job's current state.
const status = await kb.getBulkIngestionJob(jobId);
console.log(status.state, status.counts);

// Or subscribe to server-side updates.
kb.observeBulkIngestionJob(jobId).subscribe((update) => {
  console.log(update.state, update.counts);
});

// Already-finalized contexts are kept when a job is cancelled.
await kb.cancelBulkIngestionJob(jobId);

observeBulkIngestionJob() completes when the job reaches completed, failed, or cancelled. A terminal failed or cancelled state arrives as a normal emitted value rather than an error, so handle it in your next callback. The observable only errors on a transport failure.

Two details to plan for:

  • The observable is cold, so each subscription registers its own server-side subscription. Share it (for example with RxJS share()) when several consumers watch one job.
  • If the application or knowledge base is deleted while a job is in flight, the job record is purged without a final update and the observable never completes. Bound it with RxJS timeout() when that is possible.

Uploading large file sets​

Passing files directly to bulkUpsertContexts() sends them through Squid, which buffers the request in memory and therefore caps a single call at 50 files and 256 MB in total. Beyond that, mint presigned URLs and upload straight to storage, which is subject to neither cap:

Backend code
const kb = this.squid.ai().knowledgeBase('banking-knowledgebase');

// At most 500 file names per call.
const { uploads } = await kb.createBulkUploadUrls(['statement-2026-01.pdf', 'statement-2026-02.pdf']);

for (const upload of uploads) {
  await fetch(upload.uploadUrl, {
    method: 'PUT',
    body: fileBytesFor(upload.fileName),
    // Required headers are empty on S3, but Azure Blob rejects the PUT without them.
    headers: upload.requiredHeaders,
  });
}

// Reference the staged objects instead of sending bytes through Squid.
const staged = uploads.map(upload => ({
  type: 'file' as const,
  contextId: upload.fileName,
  stagedObjectKey: upload.stagedObjectKey,
}));
await kb.bulkUpsertContexts(staged);

Presigned URLs expire shortly after they are minted, so upload each wave promptly rather than minting all of them up front.

Best Practices​

  • Split large documents into focused knowledge bases by topic. This gives the agent better signal for choosing the right context.
  • Add metadata to your contexts to enable filtering at query time, reducing noise in responses.