Adding URLs and setting refresh policies

A URL source is a single public page you want Serpio to know about: a pricing page, a help article, a partner's spec sheet, a regulator's guidance. Serpio fetches it, keeps the readable text, and can check it again on a schedule. This guide covers the Add URL tab, what the fetcher will and will not accept, the Check URL for changes setting, and how change detection decides whether to re-index.

Add a URL

1. 1

   Open Add knowledge and switch tabs

   Press Add knowledge, then Add URL.

2. 2

   Enter the address

   Paste the full address into URL, including https://. Anything else is rejected with Enter a full http:// or https:// URL.

3. 3

   Optionally name and describe it

   Title · optional replaces the page's own title in your list and in the writer's evidence. How should Serpio use this? · optional is a note of up to 1,000 characters, indexed with the page so retrieval finds it for the right questions, for example: this is our current pricing; take plan names and limits from here.

4. 4

   Set usage and cadence

   Choose Use this as and Permission for writing, tick Content focuses if the page belongs to one, then pick Check URL for changes. URLs default to Weekly.

5. 5

   Press Add URL

   The modal closes and a status strip shows progress. The row reads Processing until its first version is Ready.

What the fetcher accepts

Serpio fetches URL sources itself, identifying as SerpioBot/1.0. The fetcher is deliberately narrow.

From an HTML page Serpio keeps the main content, drops navigation, footers, forms, and scripts, and preserves headings so each passage knows its section.

Check URL for changes

Checks run from a sweep every ten minutes, a few sources per brand at a time, so a due check may start some minutes late. After a failure Serpio backs off, retrying after one hour, then two, doubling up to the normal interval. Last checked and Next check in the drawer show the actual times.

How change detection works

A check does not blindly re-index. Serpio first asks the server whether the page changed, using the ETag and Last-Modified values it saved last time. If the server answers that nothing changed, Serpio records Last checked and stops. Otherwise it downloads the page, extracts the text, and compares a content hash of the text, title, headings, and description with the active version.

• Same hash: the check is recorded, Last checked updates, and no new version is created. Last changed stays where it was.
• Different hash: a new version is created and processed. The old version keeps serving until the new one is Ready, and the row shows Ready · updating in between. When it activates, Last changed moves to now and the brand's knowledge revision bumps.
• Fetch or extraction failure: the new version is marked failed, the old one stays active, and the row reads Ready · refresh failed with the error in the drawer. Nothing you already had is lost.

Check URL now and Reprocess saved content

A URL source's drawer has two buttons. Check URL now runs the fetch-and-compare cycle immediately, whatever the cadence. Reprocess saved content appears only when Serpio holds a verified full-page snapshot of the active version; it re-indexes that saved text without contacting the site, which helps after an indexing hiccup. If the snapshot is missing or the page is Removed, the button is hidden.

Only a previously saved excerpt is indexed

Sources added before the current indexing pipeline may show an amber warning beginning Only a previously saved excerpt is indexed. and a status of Legacy index · refresh needed. It means Serpio holds only a short excerpt of the page, not the verified full text. Press Check URL now. When the fresh version activates, the warning clears and the status becomes Ready.

URL sources versus the website crawler

If a page on your own site is missing from the crawl, adding it as a URL source is the fix. See Serpio's site index is not Google's index.