@metreeca/gear

Job executor and shared services for data pipelines.

@metreeca/gear runs @metreeca/flow pipelines as jobs and supplies the shared services their tasks draw on. A job names the services it needs, and the executor provides them for the duration of the run. Ready-made tasks come from the libraries built on top of it.

Job Executor: a pipeline run as a job, with every service it uses scoped to that run
Shared Services: the working space, secrets and caches a run needs, built for the run and released as it ends
Custom Bindings: a stubbed, throttled or cached implementation swapped in for a run, leaving the job untouched
Platform Bindings: service implementations for cloud platforms, one package per platform, bound in place of the local defaults

Important

Pipelines are server-side workloads targeting Node.js 22 or later, relying on facilities such as the filesystem, the process environment and fetch. The packages are not intended for the browser.

Ecosystem

@metreeca/gear focuses on job execution and shared services. Data access, content handling and model work belong to dedicated task libraries. Together they form a complete, modular toolkit for building data pipelines: every library runs under the same executor and draws on the same services.

Package	Description
@metreeca/gear	Job executor and shared services for data pipelines
@metreeca/pipe	Ready-made tasks for retrieving and persisting data from external sources
@metreeca/mime	Ready-made tasks for parsing and serialising content by media type
@metreeca/muse	Ready-made tasks and shared services for AI jobs

Installation

npm install @metreeca/gear              # job executor and shared services
npm install @metreeca/gear-<platform>   # service package, one per platform

Warning

TypeScript consumers must use "moduleResolution": "nodenext"/"node16"/"bundler" in tsconfig.json. The legacy "node" resolver is not supported.

Install the core package, then add a service package for each platform the job runs on. Service packages are self-contained leaves, each pulling in only the client libraries its own platform needs.

Package	Description
@metreeca/gear	Job executor and shared services
@metreeca/gear-gcp	Google Cloud service implementations

Usage

Note

Each package documents its own API in its README and API reference; for complete coverage, see the API reference.

A job binds the services it relies on, then drives a @metreeca/flow feed through the tasks provided by the ecosystem libraries:

import { bind, executor, service } from "@metreeca/gear";
import { createDotVault, createVault } from "@metreeca/gear/vault";
import { csv } from "@metreeca/mime-csv";
import { fetch } from "@metreeca/pipe-url";
import { pipe } from "@metreeca/flow";
import { items } from "@metreeca/flow/feeds";
import { each } from "@metreeca/flow/sinks";

await executor(
	bind(createVault, createDotVault)
)(async () => pipe(items([await service(createVault)("data-url")])
	(fetch())
	(csv())
	(each(record => console.log(record)))
));

Support

open an issue to report a problem or to suggest a new feature
start a discussion to ask a how-to question or to share an idea

License

This project is licensed under the Apache 2.0 License – see LICENSE file for details.