Skip to main content

Class: UrlLoader

Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:107

An IDocumentLoader that fetches a remote URL and delegates parsing to the appropriate registered loader based on the response Content-Type.

Supported content types​

Content-TypeDelegates to
text/htmlHtmlLoader (registry)
application/pdfPdfLoader (registry)
Everything elsePlain UTF-8 text

With fetchDocument, the answer's media type picks the loader: text/html and application/xhtml+xml go to the .html loader, application/pdf to the .pdf loader, anything else is read as UTF-8 text, and metadata.source is the answer's last address after redirects. An HTML, XHTML or PDF answer the loader source has no loader for is refused with an error, so an HTML or XHTML answer is never returned as raw markup.

Example​

const registry = new LoaderRegistry();
const urlLoader = new UrlLoader(registry);

// Register so the registry also dispatches URLs via canLoad checks.
// (Optional — UrlLoader can be used standalone too.)

if (urlLoader.canLoad('https://example.com/report.pdf')) {
const doc = await urlLoader.load('https://example.com/report.pdf');
console.log(doc.format); // 'pdf'
}

An address a person gave a server​

const loaders = {
getLoader: (extension: string) =>
extension === '.html' ? new HtmlLoader() : extension === '.pdf' ? new PdfLoader() : undefined,
};
const loader = new UrlLoader(loaders, {
fetchDocument: (url) => guardedFetch(url, { maxBytes: 2 * 1024 * 1024, deadlineMs: 10_000 }),
});
const doc = await loader.load(address);

Implements​

Implements​

Constructors​

Constructor​

new UrlLoader(registry, options?): UrlLoader

Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:123

Parameters​

registry​

LoaderSource

Answers the format-specific loader once the remote content type is known: a LoaderRegistry, or any object with getLoader(extension).

options?​

UrlLoaderOptions = {}

How the loader reads; fetchDocument replaces the global fetch.

Returns​

UrlLoader

Properties​

supportedExtensions​

readonly supportedExtensions: string[] = []

Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:114

URLs have no file extension so this array is always empty.

Routing to this loader must be performed via canLoad rather than the registry's extension-based lookup.

Implementation of​

IDocumentLoader.supportedExtensions

Methods​

canLoad()​

canLoad(source): boolean

Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:140

Returns true when source is a string that starts with http:// or https://.

Buffer sources are always rejected — raw bytes cannot be a URL.

Parameters​

source​

Absolute file path, URL string, or raw bytes.

string | Buffer

Returns​

boolean

Implementation of​

IDocumentLoader.canLoad


load()​

load(source, options?): Promise<LoadedDocument>

Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:176

Fetch source over HTTP/HTTPS and return a LoadedDocument.

The response body is buffered in memory and then handed to the appropriate sub-loader according to the Content-Type header:

  • text/html → fetched as text, passed to the HTML loader as a Buffer.
  • application/pdf → fetched as bytes, passed to the PDF loader as a Buffer.
  • Anything else → returned as plain text with format 'text' and source metadata set to the URL.

With fetchDocument, the answer it gives is handed on instead: HTML and XHTML to the HTML loader, PDF to the PDF loader, anything else as plain text, with source metadata set to the answer's last address.

Parameters​

source​

HTTP/HTTPS URL string.

string | Buffer

options?​

LoadOptions

Optional load hints forwarded to the delegated loader.

Returns​

Promise<LoadedDocument>

A promise resolving to the LoadedDocument.

Throws​

When source is a Buffer (URLs must be strings).

Throws​

When the HTTP request fails (network error or non-2xx status), or whatever fetchDocument throws.

Throws​

When the answer is a PDF and the loader source has no .pdf loader, or, with fetchDocument, HTML or XHTML and it has no .html loader.

Implementation of​

IDocumentLoader.load