Metreeca Gear
    Preparing search index...

    Function crawl

    Creates a URL graph walker or harvester.

    • Creates a URL graph walker.

      The generated task converts a feed of seed URLs into a feed of the URLs reachable from them, so that a consumer works on a whole graph of URLs while stating no more than the step from one URL to the next. URLs are emitted breadth-first in level order, every seed first, then every URL one step away from a seed, and so on, so that the first arrival at a URL is also its shallowest one.

      Crawling navigates a graph of URLs without retrieving what they stand for: retrieving a URL belongs to the pipe walker is built from and deriving results from the crawled URLs to the tasks downstream. Seeds and links are stated as URLLike values, but reach walker and the feed as parsed objects, each one the crawl's own and safe to be altered.

      Note

      • Incremental: seeds are emitted as they are pulled and reachable URLs level by level, so the feed produced runs dry as the feed drawn from and walker do; no URL reachable from a seed is emitted until the source runs dry, so the feed never completes on an endless source.
      • Materialising: every crawled URL is retained for the whole lifetime of the feed, as are the seeds and the level being crawled, so an unbounded or widely branching graph may exhaust memory.
      • Stateful: the URLs already crawled decide the ones that follow, so a task invoked per nested feed or per run crawls each independently, reaching a URL once per invocation rather than once for the feed as a whole.
      Important

      URLs are crawled at most once across the whole feed, whatever seed they are reached from, so cyclic and converging graphs are crawled without duplicates and without looping. They are matched by canonical form, so that an omitted path or an uppercase host is crawled once, while what the parser keeps apart, a trailing slash or a fragment among them, is crawled as a distinct URL.

      Parameters

      • walker: (url: URL) => Source<Awaitables<URLLike>>

        The function stating the URLs linked from a URL, none if it is a leaf

      Returns Task<URLLike, URL>

      A task converting a feed of seed URLs into a feed of the seeds and the URLs reachable from them, each as a parsed object

      Error While the feed is consumed, whatever the source reports while producing seeds, or whatever walker reports while stating the URLs linked from a URL

      TypeError While the feed is consumed, if a seed or a link cannot be parsed on its own, a relative reference among them

      const pages: Record<string, string[]> = { "/a": ["/b", "/c"], "/b": ["/d"], "/c": ["/d"], "/d": [] };

      await pipe(
        (items(["https://example.com/a"]))
        (crawl(url => pages[url.pathname]?.map(path => new URL(path, url))))
        (toArray())
      );  // the URLs of /a, /b, /c and /d, in that order

    • Creates a URL graph harvester.

      The generated task converts a feed of seed URLs into a feed of results derived from what the crawled URLs stand for, so that a consumer harvests a whole graph of URLs while stating the retrieval of a URL as a pipe of its own. Each crawled URL is handed over once, whatever the number of links converging on it, and every value it is read as is both walked and mapped from that single reading, so that the crawl is driven and harvested without reading a URL twice. Results are emitted in the level order the URLs are crawled in, the results of every seed first, then those of every URL one step away from a seed, and so on.

      Retrieval is stated as a task over a whole level rather than as a step per URL, so how many URLs are retrieved at a time is the consumer's to state with the tasks already at hand: a forked feeder retrieves several at once, an unforked one retrieves them in turn. A URL is left out of the harvest by emitting nothing for it. Seeds and links are stated as URLLike values, but reach feeder as parsed objects, each one the crawl's own and safe to be altered.

      Note

      • Incremental: the results of the seeds are emitted as feeder draws them and those of the reachable URLs level by level, so the feed produced runs dry as the feed drawn from, feeder, walker and mapper do; no URL reachable from a seed is fed until the source runs dry, so the feed never completes on an endless source.
      • Materialising: every crawled URL is retained for the whole lifetime of the feed, as for the single-step form, while what a URL stands for is released as soon as it is walked and mapped, so it is never retained across levels.
      • Stateful: the URLs already crawled decide the ones that follow, as for the single-step form.
      Important

      URLs are crawled at most once across the whole feed, whatever seed they are reached from, and matched by canonical form, as for the single-step form.

      Important

      feeder is applied to one level at a time, so state it initialises on invocation is scoped to that level rather than to the crawl, and state spanning the crawl belongs to the enclosing closure. Levels are kept apart whatever the task does, while order within a level is the task's own, so a feeder retrieving several URLs at a time harvests a level in completion order.

      Type Parameters

      • V

        The type of what a crawled URL stands for

      • R

        The type of the results derived from a crawled URL

      Parameters

      • feeder: Task<URL, V>

        The task stating what the URLs of a level stand for, emitting nothing for a URL to be crawled no further and to contribute no result

      • walker: (data: V) => Source<Awaitables<URLLike>>

        The function stating the URLs linked from what a URL stands for, none if it is a leaf

      • mapper: (data: V) => Source<R | Awaitables<R>>

        The function stating the results derived from what a URL stands for, either a single result or a sequence of them, none if it contributes no result

      Returns Task<URLLike, R>

      A task converting a feed of seed URLs into a feed of the results derived from every crawled URL

      Error While the feed is consumed, whatever the source reports while producing seeds, or whatever feeder, walker and mapper report while reading, walking and mapping a URL

      TypeError While the feed is consumed, if a seed or a link cannot be parsed on its own, a relative reference among them

      await pipe(
        (items(["https://example.com/products/"]))
        (crawl(
          fork(4, urls => urls(parse())), // the pages the URLs stand for, four retrievals at a time
          page => page.links(".pagination a"), // the index pages it paginates to
          page => page.links(".entry a") // the item links it lists
        ))
        (toArray())
      );  // the item links of every index page

    Settings
    Member Visibility