Skip to main content

Building Components

Components in Patterns are reusable graphs. They consist of one or more nodes, are given a unique slug name, and are uploaded to the publishing organization's component library. They can be simple utilities, like a SQL deduplication node, or complex flows, like an entire underwriting model and scoring system.

Quick example​

For now, component development is best done locally with help from the Patterns Devkit. Install it if you haven’t already ( pip install patterns-devkit ) and then login ( patterns login ) to connect to the Patterns platform. Sign up at studio.patterns.app if you don’t have an account.

Node files​

For our example, we’ll make a simple one node component that takes in a table, streams it and augments the records with a timestamp:

augment_with_timestamp.py
from datetime import datetime, timezone

from patterns import *

input_table = Table("input_table")
output_table = Table("output_table", "w")
timestamp_field = Parameter("timestamp_field", type=str, default="timestamp")

for record in input_table.as_stream():
    record[timestamp_field] = datetime.now(timezone.utc)
    output_table.append(record)

To make this node into a component we will make a create a graph.yml and fill in the special fields slug, version, and exposes. In addition, it is recommended to add a description_file (markdown), tags, and an icon image (which will display in graph UI and component marketplace).

graph.yml
title: Augment w/ Timestamp
slug: augment-with-timestamp
description_file: augment-with-timestamp.md
icon: augment-icon.svg
version: 0.1.0
tags:
  - utilities
exposes:
  inputs:
    - input_table
  outputs:
    - output_table
  parameters:
    - timestamp_field_name
functions:
    - node_file: augment_with_timestamp.py

We can then upload and publish our component to our organization with the devkit command patterns upload path/to/graph.yml --publish-component. (You may see “graph errors” when you upload a component that has an unconnected input or unfilled parameter, these are ok to ignore, since they will be connected by users of your component) Now if we run patterns list components we should see it included in the output.

Using components​

To use the component in a different graph, we need to include it as a node with the uses field of our graph.yml file:

graph.yml
title: Example using component
functions:
  - node_file: generate_stream.py
  - uses: my-org/augment-with-timestamp@v0
    inputs:
      input_table: my_table
    parameter_values:
      timestamp_field_name: processed_at
stores:
  - table: my_table
  - table: output_table

Advanced topics​

Schemas​

Schemas are recommended for any component where the structure of the data is known ahead of time. Schemas allow users to get clean and documented data, and allow other components to safely interoperate. Read more about how to use schemas here.