@metreeca/gear

Ready-made tasks and shared services for ETL jobs.

@metreeca/gear brings ready-made @metreeca/flow tasks for acquiring data from external sources and converting it into the values a pipeline works on. The tasks run under a pluggable job executor, which supplies the shared services they draw on.

• Ready-Made Tasks: retrieval and parsing for the usual interchange formats, chaining alongside any other task
• Shared Services: the working space, secrets and caches a run needs, built for the run and released as it ends
• Custom Bindings: a stubbed, throttled or cached implementation swapped in for a run, leaving the job untouched
• Minimal Footprint: one package per input type, each pulling in only the libraries that type needs

Installation

npm install @metreeca/gear           # job execution runtime and shared services
npm install @metreeca/gear-<type>    # task package, one per input type
Copy

Install the core package, then add a task package for each input type the pipeline handles. Task packages are self-contained leaves, each pulling in only the libraries its own input type needs.

Usage

A job binds the services it relies on, then drives a @metreeca/flow feed through the tasks provided by the input packages:

import { bind, executor, service } from "@metreeca/gear";
import { createDotVault, createVault } from "@metreeca/gear/vault";
import { csv } from "@metreeca/gear-csv";
import { fetch } from "@metreeca/gear-url";
import { pipe } from "@metreeca/flow";
import { items } from "@metreeca/flow/feeds";
import { each } from "@metreeca/flow/sinks";

await executor(
	bind(createVault, createDotVault)
)(async () => pipe(items([await service(createVault)("data-url")])
	(fetch())
	(csv())
	(each(record => console.log(record)))
));
Copy

Support

• open an issue to report a problem or to suggest a new feature
• start a discussion to ask a how-to question or to share an idea

License

This project is licensed under the Apache 2.0 License – see LICENSE file for details.