Protobuf filter¶
Metadata¶
- name:
saturnin.proto.filter
- description:
Protobuf data filter microservice
- classification:
proto/filter
- OID:
1.3.6.1.4.1.53446.1.1.0.3.3.2
- OID name:
iso.org.dod.internet.private.enterprise.firebird.butler.platform.saturnin.micro.proto.filter
- UUID:
37937949-5102-5458-a801-c327b536d51e
- facilities:
None
- API:
None
Usage¶
This microservice is a DATA FILTER that converts protobuf messages to text:
INPUT: protobuf messages
PROCESSING: uses expressions / functions evaluating data from protobuf message to filter data for output
OUTPUT: protobuf messages
Important
The namespace for functions and expressions contain only globals and module datetime
.
Configuration¶
- agent:
UUID
: Agent identification (service UUID)- logging_id:
str
: Logging ID for this component instance, see Context-based logging for details.- propagate_input_error:
bool
: When input pipe is closed with error, close output with error as well. DEFAULTTrue
.- input_pipe:
str
: Input Data Pipe Identification. REQUIRED option.- input_pipe_address:
ZMQAddress
: Input Data Pipe endpoint address. REQUIRED option.- input_pipe_mode:
SocketMode
: Input Data Pipe Mode (bind/connect). REQUIRED option.- input_pipe_format:
MIME
: Input Pipe data format specification. REQUIRED for CONNECT pipe mode.- input_batch_size:
int
: Input Pipe Data batch size. DEFAULT 50.- input_ready_schedule_interval:
int
: Input Pipe READY message schedule interval in milliseconds. See FBDP documentation for details. DEFAULT 1000.- output_pipe:
str
: Output Data Pipe Identification. REQUIRED option.- output_pipe_address:
ZMQAddress
: Output Data Pipe endpoint address. REQUIRED option.- output_pipe_mode:
SocketMode
: Output Data Pipe Mode (bind/connect). REQUIRED option.- output_pipe_format:
MIME
: Output Pipe data format specification. DEFAULTtext/plain;charset=utf-8
- output_batch_size:
int
: Output Pipe Data batch size. DEFAULT 50.- output_ready_schedule_interval:
int
: Output Pipe READY message schedule interval in milliseconds. See FBDP documentation for details. DEFAULT 1000.- include_expr:
PyExpr
: Data inclusion Python expression- include_func:
PyCallable
: Data inclusion Python function with signaturedef f(data: Any) -> bool:
.- exclude_expr:
PyExpr
: Data exclusion Python expression- exclude_func:
PyCallable
: Data exclusion Python function with signaturedef f(data: Any) -> bool:
.
Important
Input/output format is required and must be MIME_TYPE_PROTO
Input and output MIME ‘type’ params must be present and the same
At least one filter options must have a value
Only one from include and exclude methods could be defined
Example configurations that work with saturnin.core.protobuf.fblog
protobuf package:
[log-filter]
agent = 37937949-5102-5458-a801-c327b536d51e
input_pipe = pipe-1
input_pipe_address = inproc://${input_pipe}
input_pipe_mode = connect
input_pipe_format = application/x.fb.proto;type=saturnin.core.protobuf.fblog.LogEntry
;
output_pipe = pipe-2
output_pipe_address = inproc://${output_pipe}
output_pipe_mode = bind
output_pipe_format = application/x.fb.proto;type=saturnin.core.protobuf.fblog.LogEntry
include_expr = (data.timestamp.ToDatetime() > datetime.datetime.fromisoformat('2019-01-01')) and (data.facility not in [0, 10])
exclude_func =
| def foo(data: Any) -> bool:
| return data.code == 151