@metreeca/gear

Ready-made tasks and shared services for ETL jobs.

@metreeca/gear is a collection of ready-made @metreeca/flow tasks for retrieving remote or local sources and parsing the usual interchange formats, plus the job executor those tasks draw their shared services from. Key features include:

• Ready-Made Tasks: retrieval and parsing tasks chaining into a pipe alongside any other task
• Shared Services: facilities covering the surroundings a run works against, such as its working space
• Custom Bindings: any service replaced for the duration of a run, without altering the code drawing on it
• Scoped Lifecycle: service instances constructed on first use, shared across a run, and, where they implement a disposal protocol, disposed as it ends
• Minimal Footprint: install one package per input type, each pulling in only the libraries that type needs

Installation

npm install @metreeca/gear           # job execution runtime and shared services
npm install @metreeca/gear-<type>    # task packages, among those listed above
Copy

Install the core package, then add a task package for each input type the pipeline handles. These are self-contained leaf packages: each pulls in the core package transitively and only the libraries its own input type needs.

Usage

A job binds the services it relies on, then drives a @metreeca/flow feed through the tasks provided by the input packages:

import { bind, executor, service } from "@metreeca/gear";
import { createDotVault, createVault } from "@metreeca/gear/vault";
import { csv } from "@metreeca/gear-csv";
import { fetch } from "@metreeca/gear-url";
import { pipe } from "@metreeca/flow";
import { feed } from "@metreeca/flow/feeds";
import { forEach } from "@metreeca/flow/sinks";

await executor(
	bind(createVault, createDotVault)
)(async () => pipe(feed([await service(createVault)("data-url")])

	(fetch())
	(csv())

	(forEach(record => console.log(record)))
));
Copy

Support

• open an issue to report a problem or to suggest a new feature
• start a discussion to ask a how-to question or to share an idea

License

This project is licensed under the Apache 2.0 License – see LICENSE file for details.