Protobuf filter

Metadata

name:

saturnin.proto.filter

description:

Protobuf data filter microservice

classification:

proto/filter

OID:

1.3.6.1.4.1.53446.1.1.0.3.3.2

OID name:

iso.org.dod.internet.private.enterprise.firebird.butler.platform.saturnin.micro.proto.filter

UUID:

37937949-5102-5458-a801-c327b536d51e

facilities:

None

API:

None

Usage

This microservice is a DATA FILTER that converts protobuf messages to text:

  • INPUT: protobuf messages

  • PROCESSING: uses expressions / functions evaluating data from protobuf message to filter data for output

  • OUTPUT: protobuf messages

Important

The namespace for functions and expressions contain only globals and module datetime.

Configuration

agent:

UUID: Agent identification (service UUID)

logging_id:

str: Logging ID for this component instance, see Context-based logging for details.

propagate_input_error:

bool: When input pipe is closed with error, close output with error as well. DEFAULT True.

input_pipe:

str: Input Data Pipe Identification. REQUIRED option.

input_pipe_address:

ZMQAddress: Input Data Pipe endpoint address. REQUIRED option.

input_pipe_mode:

SocketMode: Input Data Pipe Mode (bind/connect). REQUIRED option.

input_pipe_format:

MIME: Input Pipe data format specification. REQUIRED for CONNECT pipe mode.

input_batch_size:

int: Input Pipe Data batch size. DEFAULT 50.

input_ready_schedule_interval:

int: Input Pipe READY message schedule interval in milliseconds. See FBDP documentation for details. DEFAULT 1000.

output_pipe:

str: Output Data Pipe Identification. REQUIRED option.

output_pipe_address:

ZMQAddress: Output Data Pipe endpoint address. REQUIRED option.

output_pipe_mode:

SocketMode: Output Data Pipe Mode (bind/connect). REQUIRED option.

output_pipe_format:

MIME: Output Pipe data format specification. DEFAULT text/plain;charset=utf-8

output_batch_size:

int: Output Pipe Data batch size. DEFAULT 50.

output_ready_schedule_interval:

int: Output Pipe READY message schedule interval in milliseconds. See FBDP documentation for details. DEFAULT 1000.

include_expr:

PyExpr: Data inclusion Python expression

include_func:

PyCallable: Data inclusion Python function with signature def f(data: Any) -> bool:.

exclude_expr:

PyExpr: Data exclusion Python expression

exclude_func:

PyCallable: Data exclusion Python function with signature def f(data: Any) -> bool:.

Important

  • Input/output format is required and must be MIME_TYPE_PROTO

  • Input and output MIME ‘type’ params must be present and the same

  • At least one filter options must have a value

  • Only one from include and exclude methods could be defined

Example configurations that work with saturnin.core.protobuf.fblog protobuf package:

[log-filter]
agent = 37937949-5102-5458-a801-c327b536d51e
input_pipe = pipe-1
input_pipe_address = inproc://${input_pipe}
input_pipe_mode = connect
input_pipe_format = application/x.fb.proto;type=saturnin.core.protobuf.fblog.LogEntry
;
output_pipe = pipe-2
output_pipe_address = inproc://${output_pipe}
output_pipe_mode = bind
output_pipe_format = application/x.fb.proto;type=saturnin.core.protobuf.fblog.LogEntry

include_expr = (data.timestamp.ToDatetime() > datetime.datetime.fromisoformat('2019-01-01')) and (data.facility not in [0, 10])

exclude_func =
    | def foo(data: Any) -> bool:
    |     return data.code == 151