Metreeca Gear
    Preparing search index...

    Function fetch

    • Creates a resource fetcher.

      The generated task converts a feed of requests into a feed of Response objects, so that a consumer draws on remote and local resources through the client the execution supplies rather than through one of its own.

      A request is given as the standard fetch() function accepts it, that is as a URL string, a URL object or a Request object. A response is emitted as soon as its head is received, with the body left unread for the consumer to draw or discard; a response carrying no body is emitted all the same.

      Exchanges are routed through the fetch client resolved from the enclosing execution, so that the transport is chosen when the task is run rather than by the task itself: binding createFetch routes every exchange through a throttled, cached or stubbed client, without altering the task, while an unbound execution falls back on the standard fetch function.

      Exchanges are sent under desktop browser User-Agent and Accept fields, so that sites serving unattended clients differently, or refusing them altogether, are scraped as a browser would be; a request already stating one of these fields keeps its own value, as do the ones stated by middlewares.

      Content coding is left to the transport, which states the codings it accepts and decodes the body before it reaches the consumer, so that no Accept-Encoding field is stated here and no body is handed over still compressed.

      Every exchange is reported to the log as it is performed, so that a run leaves a trace of the resources it drew on and of the ones it was denied; requests are reported as they are stated, before middlewares are given a chance to alter them.

      Note

      • Incremental: each response is emitted as soon as its head is received, so the feed produced runs dry as the feed drawn from does and an endless source is read as long as it is consumed.
      • Streaming: responses are drawn one at a time and handed over with the body unread, none retained, so resources of any size are handled without holding them in memory, as long as each body is read or cancelled before the next response is drawn.
      • Stateless: every request is exchanged on its own, so the outcome is unaffected by how the feed is split across nested feeds or runs, whatever state middlewares and the resolved client carry across exchanges.
      Warning

      A request stating a URL that is not absolute, or malformed in any other way, is dropped before it is sent, as is a response stating an unsuccessful status; both are reported to the log, leaving the feed to run to completion.

      Parameters

      • ...middlewares: readonly Middleware[]

        The middlewares to be layered over the resolved fetch client, in request processing order

      Returns Task<string | URL | Request, Response>

      A task converting a feed of requests into a feed of responses

      Error While the feed is consumed, if no execution is running, as the fetch client is resolved from the enclosing one

      Error While the feed is consumed, whatever the source reports while producing requests, or whatever the exchange reports while connecting to a resource or receiving its response

      fetch()

    Settings
    Member Visibility