Skip to main content

Generic Site Ingester

Sync any website's pages into a knowledge base by discovering them from its sitemap.xml​

The Generic Site Ingester connector's functionality​

The Generic Site Ingester connector reads the pages of a website into a knowledge base, enabling Squid AI agents to answer questions using that site's content:

  • Squid discovers pages from the site's sitemap.xml, following the sitemaps.org protocol. Sitemap index files (a sitemap that only lists other sitemaps) are not supported. Point the connector at a sitemap that lists pages directly.
  • Each page is fetched, converted to Markdown, and indexed for semantic search.
  • Squid syncs the site once a day. Pages whose sitemap lastmod timestamp hasn't changed since the last sync are skipped, and pages no longer listed in the sitemap are removed from the knowledge base.
  • Pages that resolve to a non-English locale (via hreflang alternates or a locale segment in the URL, such as /fr/...) are skipped. Pages with no language signal at all are kept.
  • AI agents connected to the connector can perform semantic searches over the indexed content, so users can ask questions like "How do I reset my password?" and get answers sourced from the site.

The connector provides read-only access: agents can search and reference the site's content, but cannot modify it.

Rendering JavaScript-heavy sites

By default, pages are fetched with a plain HTTP request, which doesn't run client-side JavaScript. If a site's content is rendered client-side (a single-page app, or content that loads after the initial page load), enable Use Browser Rendering so each page is rendered in a headless browser before extraction. This is slower and more resource-intensive per page, so leave it off unless pages are missing content: most server-rendered sites and docs/help centers don't need it.

Adding the Generic Site Ingester connector to your Squid application​

  1. Navigate to the Connectors tab in the Squid Console.

  2. Click Available Connectors.

  3. Find the Generic Site Ingester connector, and select Add Connector.

  4. Provide the following configuration details:

Connector ID: A string that uniquely identifies the connector in your code.

Type: How pages are discovered. Only Sitemap is supported today.

Sitemap URL: The URL of the site's sitemap.xml, such as https://example.com/sitemap.xml.

Use Browser Rendering: Optional. Renders each page in a headless browser before extracting its content, so JavaScript-rendered content is captured. Leave off unless pages are missing content.

note

After adding the connector, allow some time for the initial sync to complete. A large sitemap can take a while to fully index.

Triggering a manual sync​

The connector syncs automatically once a day, and immediately after it's created or its Sitemap URL is changed. To trigger a sync on demand, call its webhook:

https://{your-app-id}.{your-region}.squid.cloud/webhooks/syncGenericSiteIngester?integrationId={your-connector-id}

integrationId is always required. Add the following optional query parameters:

  • fullResync=true: Replays every page in the sitemap, ignoring lastmod timestamps and previous failures. Useful after a sync issue, or when you want to force a full reindex.
  • dryRun=true: Reports what a sync would do (pages to sync, skip, or delete) without fetching any page content or changing the knowledge base.

Changing the Sitemap URL on an existing connector resets its sync state, so the next sync discovers and syncs every page again, as if it were the connector's first sync.

Using the Generic Site Ingester connector in your application​

No-code Studio​

  1. Navigate to the Studio tab in the Squid Console.

  2. Click Create AI Agent.

  3. Provide an agent ID and description, such as "docs-agent" and "This agent answers questions using our website's content".

  4. Click Add Abilities.

  5. Expand the Generic Site Ingester entry and select the connector you created earlier.

  6. Give a description for how the agent should use this connection, such as "Call this when a user asks a question that may be answered by content on our website."

  7. Click on Test and try asking the agent a question about content on the site.

Using the Squid SDK​

If you are creating your AI agent with the Squid SDK, connect the Generic Site Ingester connector by adding it to the connectedIntegrations option of the ask() function:

Backend code
await this.squid
  .ai()
  .agent('AGENT_ID')
  .ask('How do I reset my password?', {
    connectedIntegrations: [
      {
        integrationId: 'GENERIC_SITE_INGESTER_CONNECTOR_ID',
        integrationType: 'generic_site_ingester',
        description: 'Call this connector to search our website content',
      },
    ],
  });

To learn more about interacting with an agent from the SDK, view the AI agent documentation.