Creates a task interleaving several runs of another task over the same items.
Runs draw from the same source, each item going to exactly one of them, processed through that run's own invocation
of task and reported as soon as it is ready, so a slow item never holds back the others. runs caps the items
being processed at the same time, which is also how far ahead of the consumer items are drawn from the source; an
uncapped fork opens a run for every item the source delivers instead, trading that ceiling for a throughput bounded
only by the source.
This is the concurrency knob for the work a task performs: flat() draws the feeds a task opens one at a time and
join() opens every one of them at once, so neither caps the work in flight and the window belongs around the task
instead. Runs are interleaved on the event loop rather than executed in parallel, so a task blocking it gains
nothing from being forked.
Warning
Incremental: results are emitted as the runs settle, so the reported feed runs dry as the feed drawn from
and task do.
Materialising: an uncapped fork holds a run for every item drawn, so a source delivering faster than the
runs settle may exhaust memory; cap the items in flight with runs.
Stateless: the fork carries no state across runs, whatever task carries within each.
Warning
Output order is not preserved: results interleave and overtake each other according to how long every item takes.
Caution
runs caps the items in flight, not the rate at which they are drawn: a source that never runs dry keeps every
run permanently busy, so pacing belongs inside task.
Caution
A stateful task never sees the whole feed. task is invoked once per run, so state it initialises on invocation
is scoped to that run and decides on the items that run happens to draw: a quota is granted to each run, a
deduplication spans one alone, an ordering covers only what that run drew. State captured in the enclosing
closure is shared across the runs instead, and reached concurrently.
Fork a stateful task only where its outcome is sound on part of the items. Where the state belongs to one item
rather than to the feed, open a pipe per item with map() and collapse the pipes with join(), as
join(map(…)), which keeps the state scoped to its item while still drawing from every one of them at once.
Note
Runs failing while the consumer is idle report their error when the feed is next advanced, rather than escaping as
unhandled rejections.
Note
Every run and the source are closed when the feed is exhausted, fails or is closed early, waiting for the work
already in flight to settle first, so a source idling between items delays it; failures reported while closing are
suppressed.
Type Parameters
V
The type of items drawn from the feed
R
The type of items reported by task
Parameters
runs: number
The number of concurrent runs; 0 opens a run on demand, capping none, while values below 0 are treated
as 1, that is, as sequential processing
Creates a task interleaving several runs of another task over the same items.
Runs draw from the same source, each item going to exactly one of them, processed through that run's own invocation of
taskand reported as soon as it is ready, so a slow item never holds back the others.runscaps the items being processed at the same time, which is also how far ahead of the consumer items are drawn from the source; an uncapped fork opens a run for every item the source delivers instead, trading that ceiling for a throughput bounded only by the source.This is the concurrency knob for the work a task performs:
flat()draws the feeds a task opens one at a time andjoin()opens every one of them at once, so neither caps the work in flight and the window belongs around the task instead. Runs are interleaved on the event loop rather than executed in parallel, so a task blocking it gains nothing from being forked.taskdo.runs.taskcarries within each.Output order is not preserved: results interleave and overtake each other according to how long every item takes.
runscaps the items in flight, not the rate at which they are drawn: a source that never runs dry keeps every run permanently busy, so pacing belongs insidetask.A stateful task never sees the whole feed.
taskis invoked once per run, so state it initialises on invocation is scoped to that run and decides on the items that run happens to draw: a quota is granted to each run, a deduplication spans one alone, an ordering covers only what that run drew. State captured in the enclosing closure is shared across the runs instead, and reached concurrently.Fork a stateful task only where its outcome is sound on part of the items. Where the state belongs to one item rather than to the feed, open a pipe per item with
map()and collapse the pipes withjoin(), asjoin(map(…)), which keeps the state scoped to its item while still drawing from every one of them at once.Runs failing while the consumer is idle report their error when the feed is next advanced, rather than escaping as unhandled rejections.
Every run and the source are closed when the feed is exhausted, fails or is closed early, waiting for the work already in flight to settle first, so a source idling between items delays it; failures reported while closing are suppressed.