apigenie-generator server faststream
Tier 2 · Release 0.12.0: Level 1 ✓ · Level 2 ✓ · Level 3 ✓ · Verification
Generate a Python FastStream application of the Kafka, AMQP (RabbitMQ) and MQTT channels of an AsyncAPI document.
Usage
apigenie-generator server faststream [...OPTIONS] INPUT_FILE
Description
The input must be an AsyncAPI 3.x document. Documents referencing other files are bundled first.
The application is the one the document describes: it receives the messages of receive
operations and sends those of send operations.
- one broker per server:
KafkaBrokerforkafkaandkafka-secureservers (aiokafka),RabbitBrokerforamqpandamqps(AMQP 0.9.1) andMQTTBrokerformqtt,mqttsandsecure-mqtt(MQTT 5 or 3.1.1). A broker connects to the URL in an environment variable named after its server (PRODUCTION_KAFKA_URL), or to the server’s address with the default values of its variables - a subscriber per receive operation,
@broker.subscriber(...), which calls a handler you write - a publisher per send operation,
@broker.publisher(...), and anasyncfunction publishing through it with typed arguments - Pydantic v2 models of the payloads and headers
from warehouse_events import models
from warehouse_events.publishers import publish_audit, request_quote
await publish_audit(models.AuditEntry(action="login"), region="eu")
quote = await request_quote(models.QuoteRequest(sku="B"), timeout=5)
Run the application with faststream run <package>.app:app.
Channels and bindings
| Document | Generated code |
|---|---|
Channel address, Kafka binding topic |
the topic; with parameters a pattern= subscription and Path() parameters |
Channel parameters with enum |
Literal[...] types of the Path() parameters and of the publish functions’ arguments |
Channel parameters with a location in the payload ($message.payload#/destinationCountry) |
optional arguments of the publish functions, read from the message when not given |
Kafka message binding key (a string schema) |
a key: str | None argument of the publish functions, sent as the record’s key |
Kafka operation binding groupId (const or a single enum value), also from operation traits |
group_id= of the subscriber |
AMQP binding is: queue |
a RabbitQueue (name, durable, exclusive, auto_delete) the operations publish to and consume from |
AMQP binding is: routingKey with an exchange |
a RabbitExchange in brokers.py; the address is the routing key, with parameters a topic pattern and Path() parameters. Subscribers bind a queue named after the operation |
AMQP vhost |
virtualhost= of the broker |
AMQP operation binding priority, expiration, deliveryMode: 2, mandatory, userId, timestamp |
the publisher’s priority, expiration (in seconds), persist, mandatory, user_id; a timestamp per message |
AMQP message binding contentEncoding (gzip, deflate), messageType |
published payloads compressed and sent with content_encoding= and message_type=; received ones decompressed by the subscriber’s decoder. The messages of an operation must agree on them, or none is set (a warning) |
| MQTT topic parameters (whole topic levels) | Path() parameters |
MQTT operation binding qos, retain |
qos=, retain= |
MQTT message binding responseTopic, correlationData |
a request with a reply address publishes with reply_to= (the MQTT 5 response topic) and correlation_id= (its correlation data) as well as the headers |
MQTT server binding clientId, cleanSession, keepAlive, sessionExpiryInterval, lastWill |
the broker’s client_id, clean_session, keepalive, session_expiry_interval, will |
Messages
A JSON payload becomes a Pydantic model, a text/* one str and any other content type
bytes. The models reject the properties their schemas don’t allow (extra="forbid" for
additionalProperties: false) and check the keys and the sizes of maps (propertyNames,
patternProperties, minProperties, maxProperties), and they carry the examples of the
messages’ payloads in json_schema_extra, for FastStream’s AsyncAPI page. The docstring of each
publish function lists its messages, with their titles and summaries. A payload referencing a schema of components/schemas is the model of that schema;
other payloads and headers get a model named after their message (OrderCreated,
OrderCreatedHeaders). The headers of message traits are merged into the message’s own as
AsyncAPI specifies (JSON Merge Patch: the message over its traits, later traits over earlier
ones).
Subscribers take the headers as typed Header() parameters and pass them to the handlers as
keyword arguments; publish functions take a headers model. When several messages of a channel
can be told apart by a property with a distinct constant value (type: { const: created }), the
payload is a discriminated union and every message has a handler of its own
(consume_orders_order_created); otherwise the handler of the operation gets the union.
Avro payloads
A message received from Kafka with an Avro payload
(schemaFormat: application/vnd.apache.avro;version=1.9.0) is a Pydantic model of its record in
avro.py, the same as generate client python’s, with enums of its enum types. The subscriber’s
decoder reads the record with fastavro: the Confluent framing
(a magic byte and the schema id) in the payload, or the schema id in a header when the message’s
Kafka binding says so (schemaIdLocation: header). The schema in the document is the writer’s
schema: the registry isn’t asked. A root schema that isn’t a record, records with decimal
fields and unions other than ["null", T] are skipped with a warning.
Avro messages sent by the application, or received from other brokers, or received together with other messages by the same operation, are skipped with a warning: FastStream’s Kafka publishers send headers as text, which the binary schema id header isn’t.
Security
security.py has a function per security scheme of the servers returning its FastStream
security object, and a function per broker, <broker>_security(), that brokers.py passes as
security=:
| Scheme | Security object |
|---|---|
Kafka scramSha512, scramSha256, plain (and userPassword) |
SASLScram512, SASLScram256, SASLPlaintext |
X509 |
RabbitExternalAuth on AMQP servers, BaseSecurity with a client certificate elsewhere |
AMQP and MQTT userPassword, plain |
SASLPlaintext |
MQTT openIdConnect, oauth2 |
SASLPlaintext with the token as the password |
The broker’s function reads the scheme’s key from an environment variable named after the server
(PRODUCTION_KAFKA_SECURITY=kafkaScram) and its credentials from variables named after the
scheme: KAFKA_SCRAM_USERNAME and KAFKA_SCRAM_PASSWORD, <KEY>_CERT, <KEY>_KEY and
<KEY>_CA for certificates, <KEY>_TOKEN and an optional <KEY>_USERNAME for tokens.
Without the variable, the broker connects without security. TLS servers (kafka-secure, amqps,
mqtts, secure-mqtt) verify the server with the system’s CA certificates. The README lists the
variables of each broker and scheme.
Request/reply
- A send operation with a reply channel: its function publishes the request and returns the
reply, waiting at most
timeoutseconds. A subscriber of the reply channel hands replies to the waiting requests by correlation id: the message’scorrelationId(a header, which the function fills in when the request has none, or a payload property), or FastStream’s correlation id when the document declares none. FastStream’sbroker.request()isn’t used: Kafka doesn’t support it outside the test client, and RabbitMQ’s waits onamq.rabbitmq.reply-toinstead of the reply channel of the document. - A send operation with a reply address in a header (
$message.header#/replyTo): its function takes areply_toargument and sends it in that header. - A receive operation with a reply channel: the handler returns the reply, which is published to the reply channel with the request’s correlation id header.
- A receive operation with a reply address in a header: the handler returns the reply, which is published to the address in the request’s header.
Output
pyproject.toml faststream[cli,…], pydantic; `pip install -e .`
README.md how to run, the environment variables, what isn't generated
<package>/app.py the FastStream application
<package>/brokers.py the brokers and AMQP exchanges
<package>/subscribers.py
<package>/publishers.py
<package>/handlers.py the handlers you write
<package>/models.py Pydantic models (shared with apigenie-generator client python)
<package>/security.py the security objects of the brokers, with security schemes
<package>/avro.py the Avro records, with Avro payloads (and _avro.py, fastavro)
<package>/_models.py, _runtime.py, __init__.py, py.typed
handlers.py is written only when it doesn’t exist: it holds your code. After regenerating, add
the handlers of new operations to it; mypy reports the missing ones.
Not generated
Reported as info lines or warnings:
-
channels of WebSocket and HTTP servers, which FastStream has no broker for
-
messages with payloads in other non-JSON Schema formats, Avro messages the application sends (see “Avro payloads”), and operations left without messages
-
the security schemes FastStream has no security object for (
apiKey,httpApiKey,http,scramShaon other protocols, …).symmetricEncryptionandasymmetricEncryptionencrypt the messages, not the connection: an info line says that the handlers and publishers must do it -
AMQP message bindings the messages of an operation disagree on, and content encodings other than
gzipanddeflate -
servers no generated operation uses; a channel with several servers is served by its first one (choose others with the
serversoption) -
reply addresses in payloads, AMQP
ccandbcc, MQTT topic parameters inside a topic level
The MQTT message binding’s payloadFormatIndicator and messageExpiryInterval aren’t applied:
FastStream’s MQTT publishers don’t set them.
Extensions
| Extension | Effect |
|---|---|
x-apigenie.codegen.name |
on a component schema: the name of its model class; on an operation: the name of its function (in snake_case) |
x-enum-varnames, x-enumNames |
names of the members of an enum |
x-apigenie.codegen.disabled |
true on a channel or an operation leaves it out (with the operations of a channel left out) |
The extensions can live in an overlay for this target instead of in the document, e.g.
faststream.overlay.yaml:
overlay: 1.0.0
info:
title: FastStream names
version: 1.0.0
actions:
- target: $.operations.publishOrder
update:
x-apigenie:
codegen:
name: announce_order
- target: $.components.schemas.OrderStatus
update:
x-enum-varnames: [CREATED, PAID, SHIPPED]
apigenie-generator server faststream asyncapi.yaml --overlay faststream.overlay.yaml -o out
Options
-c, –config
Path to the configuration file to use. See the “Configuration” section for more information.
–overlay OVERLAY
Path to an OpenAPI Overlay (1.0 to 1.2) applied to the input before generating, e.g. one setting the
extensions of the “Extensions” section. Can be repeated: the overlays apply in order, after those
listed under commands.generate.server_faststream.overlay in the configuration file (with paths
relative to it). Warnings and errors about a value an overlay set point to the overlay. The overlays
apply to the AsyncAPI document bundled with the files it references.
–strict
Makes an overlay action whose target selects nothing an error (exit status 2, nothing is generated)
instead of a warning (OVERLAY:LINE:COL: action #N (TARGET) selects no nodes.).
–check
Writes nothing: compares what the generator would write with the output directory, prints the files that would be created or changed, and exits with 1 when there are any (since 0.12.0). See Check that generated code is up to date.
-o, –output
Directory where the package is generated.
–package-name
Name of the generated distribution; the Python package is its snake_case form. Defaults to the title of the API.
–package-version
Version of the generated package. Defaults to the version of the API.
Configuration
version: '1.0'
commands:
generate:
server_faststream:
servers: [production-kafka, customs-broker]
enum_style: enum
field_prefix: field_
class_overrides:
ShipmentId: ShipmentKey
| Option | Description |
|---|---|
servers |
Keys of the servers to generate brokers for; all Kafka, AMQP and MQTT servers by default |
enum_style |
enum (default): class Status(str, Enum) for enums of component schemas; literal: Literal[...] everywhere |
field_prefix |
Prefix of field names that aren’t valid identifiers or clash with Pydantic, e.g. field_json (default field_) |
class_overrides |
Class names of component schemas, by schema name |
Verification
The codegen conformance suite runs this target on the Carpet Express AsyncAPI fixture. Level 1 compares the output with its snapshots and checks each feature, Level 2 runs the language's type checker and linter, and Level 3 runs the generated code in contract tests against the fixture. All targets.
This target's support tier is Tier 2. It goes through all three levels before every release, like Tier 1. It doesn't aim to support every feature: a feature of the fixture can stay unsupported, with the reason its manifest gives.
| Results | Level 1 | Level 2 | Level 3 |
|---|---|---|---|
| Release 0.12.0 | ✓ snapshots of 3.0; 50 supported, 0 not yet | ✓ pass (python 3.12.11 faststream 0.7.7 pydantic 2.13.5 mypy 2.4.0, ruff 0.16.10) | ✓ 10 of 10 operations pass (python 3.12.11 faststream 0.7.7 pydantic 2.13.5 mypy 2.4.0, ruff 0.16.10) |
| Development version | ✓ snapshots of 3.0; 50 supported, 0 not yet | runs before a release | runs before a release |