Class: UrlLoader
Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:107
An IDocumentLoader that fetches a remote URL and delegates parsing
to the appropriate registered loader based on the response Content-Type.
Supported content types
| Content-Type | Delegates to |
|---|---|
text/html | HtmlLoader (registry) |
application/pdf | PdfLoader (registry) |
| Everything else | Plain UTF-8 text |
With fetchDocument, the answer's media type picks the loader:
text/html and application/xhtml+xml go to the .html loader,
application/pdf to the .pdf loader, anything else is read as UTF-8
text, and metadata.source is the answer's last address after redirects.
An HTML, XHTML or PDF answer the loader source has no loader for is refused
with an error, so an HTML or XHTML answer is never returned as raw markup.
Example
const registry = new LoaderRegistry();
const urlLoader = new UrlLoader(registry);
// Register so the registry also dispatches URLs via canLoad checks.
// (Optional — UrlLoader can be used standalone too.)
if (urlLoader.canLoad('https://example.com/report.pdf')) {
const doc = await urlLoader.load('https://example.com/report.pdf');
console.log(doc.format); // 'pdf'
}
An address a person gave a server
const loaders = {
getLoader: (extension: string) =>
extension === '.html' ? new HtmlLoader() : extension === '.pdf' ? new PdfLoader() : undefined,
};
const loader = new UrlLoader(loaders, {
fetchDocument: (url) => guardedFetch(url, { maxBytes: 2 * 1024 * 1024, deadlineMs: 10_000 }),
});
const doc = await loader.load(address);
Implements
Implements
Constructors
Constructor
new UrlLoader(
registry,options?):UrlLoader
Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:123
Parameters
registry
Answers the format-specific loader once the remote
content type is known: a LoaderRegistry, or any object
with getLoader(extension).
options?
UrlLoaderOptions = {}
How the loader reads; fetchDocument replaces the
global fetch.
Returns
UrlLoader
Properties
supportedExtensions
readonlysupportedExtensions:string[] =[]
Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:114
URLs have no file extension so this array is always empty.
Routing to this loader must be performed via canLoad rather than the registry's extension-based lookup.
Implementation of
IDocumentLoader.supportedExtensions
Methods
canLoad()
canLoad(
source):boolean
Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:140
Returns true when source is a string that starts with http:// or
https://.
Buffer sources are always rejected — raw bytes cannot be a URL.
Parameters
source
Absolute file path, URL string, or raw bytes.
string | Buffer
Returns
boolean
Implementation of
load()
load(
source,options?):Promise<LoadedDocument>
Defined in: packages/agentos/src/cognition/memory/io/ingestion/UrlLoader.ts:176
Fetch source over HTTP/HTTPS and return a LoadedDocument.
The response body is buffered in memory and then handed to the appropriate
sub-loader according to the Content-Type header:
text/html→ fetched as text, passed to the HTML loader as aBuffer.application/pdf→ fetched as bytes, passed to the PDF loader as aBuffer.- Anything else → returned as plain text with format
'text'andsourcemetadata set to the URL.
With fetchDocument, the answer it gives is handed on instead: HTML and
XHTML to the HTML loader, PDF to the PDF loader, anything else as plain
text, with source metadata set to the answer's last address.
Parameters
source
HTTP/HTTPS URL string.
string | Buffer
options?
Optional load hints forwarded to the delegated loader.
Returns
Promise<LoadedDocument>
A promise resolving to the LoadedDocument.
Throws
When source is a Buffer (URLs must be strings).
Throws
When the HTTP request fails (network error or non-2xx
status), or whatever fetchDocument throws.
Throws
When the answer is a PDF and the loader source has no
.pdf loader, or, with fetchDocument, HTML or XHTML and
it has no .html loader.