Storing data

Table

Table nodes are storage nodes that are similar to standard database tables, but provide a layer of abstraction that enables data versioning, retention, and cross-platform support. They are attached to a specific database and create actual database tables based on the operations of Python and SQL nodes.

Schema

All tables have an associated Schema that is represented in a database engine agnostic format with a Common Model schema. Schemas define both the structure and data types of a table as well as semantic information that can be used by Patterns nodes and components to provide smart default behavior. An example schema:

OrderSchema.yml

name: Order
description: E-commerce order
unique_on:
  - id
field_roles:
  created_ordering: created_at
  updated_ordering: updated_at
  strictly_monotonic_ordering: id
immutable: false
fields:
  id:
    type: Integer
  customer_id:
    type: Text
  amount:
    type: Decimal(12, 2)
  created_at:
    type: DateTime
  updated_at:
    type: DateTime

info

N.B. field_roles are used by components and the Patterns protocol to provided semantic information for automatic behavior. A primary use case for field roles is to provide the default stream ordering of a table (i.e. used when calling table.as_stream() without an order_by argument), which uses strictly_montonic_ordering by default and falls back to created_ordering otherwise. If neither are defined, calling as_stream() with no order_by argument will result in an error.