The generated task converts a feed of requests into a feed of
Response objects, so that a consumer draws on remote and
local resources through the client the execution supplies rather than through one of its own.
A request is given as the standard fetch() function accepts it, that is as a URL string, a
URL object or a
Request object. A response is emitted as soon as its head
is received, with the body left unread for the consumer to draw or discard; a response carrying no body is emitted
all the same.
Exchanges are routed through the fetch client of the execution enclosing the
call to this factory, so the transport is chosen by the job rather than by the task. Binding createFetch
routes every exchange through a throttled, cached or stubbed client without altering the task. An execution with no
binding falls back on the standard fetch function.
Exchanges are sent with desktop browser User-Agent and Accept fields. Sites that serve unattended clients
differently, or refuse them altogether, are thus scraped as a browser would be. A request that already states one of
these fields keeps its own value, as do the values stated by middlewares.
Content coding is negotiated by the fetch client, which decodes response bodies before they reach the consumer. The
task states no Accept-Encoding field of its own.
Every exchange is logged as it is performed, so a run leaves a trace of the resources it drew on and of the ones it
was denied. Requests are logged as stated, before middlewares can alter them.
Note
Incremental: each response is emitted as soon as its head is received, so the feed produced runs dry when
the source does, and an endless source is read for as long as the feed is consumed.
Streaming: responses are drawn one at a time and handed over with the body unread, none retained, so
resources of any size are handled without holding them in memory, as long as each body is read or cancelled
before the next response is drawn.
Stateless: the task keeps no state across requests, so the outcome is unaffected by how the feed is split
across nested feeds or runs; any state carried across exchanges belongs to middlewares and the fetch client.
Warning
A request stating a URL that is not absolute, or malformed in any other way, is dropped before it is sent, as is
a response stating an unsuccessful status; both are reported to the log, leaving the feed to run to completion.
Parameters
...middlewares: readonlyMiddleware[]
The middlewares to be layered over the fetch client, in request processing order
A task converting a feed of requests into a feed of responses
Throws
Error If no execution is running, as the fetch client is resolved when the task is created
Throws
Error While the feed is consumed, whatever the source reports while producing requests, or
whatever the exchange reports while connecting to a resource or receiving its response
Creates a resource fetcher.
The generated task converts a feed of requests into a feed of Response objects, so that a consumer draws on remote and local resources through the client the execution supplies rather than through one of its own.
A request is given as the standard
fetch()function accepts it, that is as a URL string, a URL object or a Request object. A response is emitted as soon as its head is received, with the body left unread for the consumer to draw or discard; a response carrying no body is emitted all the same.Exchanges are routed through the fetch client of the execution enclosing the call to this factory, so the transport is chosen by the job rather than by the task. Binding createFetch routes every exchange through a throttled, cached or stubbed client without altering the task. An execution with no binding falls back on the standard
fetchfunction.Exchanges are sent with desktop browser
User-AgentandAcceptfields. Sites that serve unattended clients differently, or refuse them altogether, are thus scraped as a browser would be. A request that already states one of these fields keeps its own value, as do the values stated bymiddlewares.Content coding is negotiated by the fetch client, which decodes response bodies before they reach the consumer. The task states no
Accept-Encodingfield of its own.Every exchange is logged as it is performed, so a run leaves a trace of the resources it drew on and of the ones it was denied. Requests are logged as stated, before
middlewarescan alter them.middlewaresand the fetch client.A request stating a URL that is not absolute, or malformed in any other way, is dropped before it is sent, as is a response stating an unsuccessful status; both are reported to the log, leaving the feed to run to completion.