apigenie-generator server faststream

Tier 2 · Release 0.12.0: Level 1 ✓ · Level 2 ✓ · Level 3 ✓ · Verification

Generate a Python FastStream application of the Kafka, AMQP (RabbitMQ) and MQTT channels of an AsyncAPI document.

Usage

apigenie-generator server faststream [...OPTIONS] INPUT_FILE

Description

The input must be an AsyncAPI 3.x document. Documents referencing other files are bundled first.

The application is the one the document describes: it receives the messages of receive operations and sends those of send operations.

  • one broker per server: KafkaBroker for kafka and kafka-secure servers (aiokafka), RabbitBroker for amqp and amqps (AMQP 0.9.1) and MQTTBroker for mqtt, mqtts and secure-mqtt (MQTT 5 or 3.1.1). A broker connects to the URL in an environment variable named after its server (PRODUCTION_KAFKA_URL), or to the server’s address with the default values of its variables
  • a subscriber per receive operation, @broker.subscriber(...), which calls a handler you write
  • a publisher per send operation, @broker.publisher(...), and an async function publishing through it with typed arguments
  • Pydantic v2 models of the payloads and headers
from warehouse_events import models
from warehouse_events.publishers import publish_audit, request_quote

await publish_audit(models.AuditEntry(action="login"), region="eu")
quote = await request_quote(models.QuoteRequest(sku="B"), timeout=5)

Run the application with faststream run <package>.app:app.

Channels and bindings

Document Generated code
Channel address, Kafka binding topic the topic; with parameters a pattern= subscription and Path() parameters
Channel parameters with enum Literal[...] types of the Path() parameters and of the publish functions’ arguments
Channel parameters with a location in the payload ($message.payload#/destinationCountry) optional arguments of the publish functions, read from the message when not given
Kafka message binding key (a string schema) a key: str | None argument of the publish functions, sent as the record’s key
Kafka operation binding groupId (const or a single enum value), also from operation traits group_id= of the subscriber
AMQP binding is: queue a RabbitQueue (name, durable, exclusive, auto_delete) the operations publish to and consume from
AMQP binding is: routingKey with an exchange a RabbitExchange in brokers.py; the address is the routing key, with parameters a topic pattern and Path() parameters. Subscribers bind a queue named after the operation
AMQP vhost virtualhost= of the broker
AMQP operation binding priority, expiration, deliveryMode: 2, mandatory, userId, timestamp the publisher’s priority, expiration (in seconds), persist, mandatory, user_id; a timestamp per message
AMQP message binding contentEncoding (gzip, deflate), messageType published payloads compressed and sent with content_encoding= and message_type=; received ones decompressed by the subscriber’s decoder. The messages of an operation must agree on them, or none is set (a warning)
MQTT topic parameters (whole topic levels) Path() parameters
MQTT operation binding qos, retain qos=, retain=
MQTT message binding responseTopic, correlationData a request with a reply address publishes with reply_to= (the MQTT 5 response topic) and correlation_id= (its correlation data) as well as the headers
MQTT server binding clientId, cleanSession, keepAlive, sessionExpiryInterval, lastWill the broker’s client_id, clean_session, keepalive, session_expiry_interval, will

Messages

A JSON payload becomes a Pydantic model, a text/* one str and any other content type bytes. The models reject the properties their schemas don’t allow (extra="forbid" for additionalProperties: false) and check the keys and the sizes of maps (propertyNames, patternProperties, minProperties, maxProperties), and they carry the examples of the messages’ payloads in json_schema_extra, for FastStream’s AsyncAPI page. The docstring of each publish function lists its messages, with their titles and summaries. A payload referencing a schema of components/schemas is the model of that schema; other payloads and headers get a model named after their message (OrderCreated, OrderCreatedHeaders). The headers of message traits are merged into the message’s own as AsyncAPI specifies (JSON Merge Patch: the message over its traits, later traits over earlier ones).

Subscribers take the headers as typed Header() parameters and pass them to the handlers as keyword arguments; publish functions take a headers model. When several messages of a channel can be told apart by a property with a distinct constant value (type: { const: created }), the payload is a discriminated union and every message has a handler of its own (consume_orders_order_created); otherwise the handler of the operation gets the union.

Avro payloads

A message received from Kafka with an Avro payload (schemaFormat: application/vnd.apache.avro;version=1.9.0) is a Pydantic model of its record in avro.py, the same as generate client python’s, with enums of its enum types. The subscriber’s decoder reads the record with fastavro: the Confluent framing (a magic byte and the schema id) in the payload, or the schema id in a header when the message’s Kafka binding says so (schemaIdLocation: header). The schema in the document is the writer’s schema: the registry isn’t asked. A root schema that isn’t a record, records with decimal fields and unions other than ["null", T] are skipped with a warning.

Avro messages sent by the application, or received from other brokers, or received together with other messages by the same operation, are skipped with a warning: FastStream’s Kafka publishers send headers as text, which the binary schema id header isn’t.

Security

security.py has a function per security scheme of the servers returning its FastStream security object, and a function per broker, <broker>_security(), that brokers.py passes as security=:

Scheme Security object
Kafka scramSha512, scramSha256, plain (and userPassword) SASLScram512, SASLScram256, SASLPlaintext
X509 RabbitExternalAuth on AMQP servers, BaseSecurity with a client certificate elsewhere
AMQP and MQTT userPassword, plain SASLPlaintext
MQTT openIdConnect, oauth2 SASLPlaintext with the token as the password

The broker’s function reads the scheme’s key from an environment variable named after the server (PRODUCTION_KAFKA_SECURITY=kafkaScram) and its credentials from variables named after the scheme: KAFKA_SCRAM_USERNAME and KAFKA_SCRAM_PASSWORD, <KEY>_CERT, <KEY>_KEY and <KEY>_CA for certificates, <KEY>_TOKEN and an optional <KEY>_USERNAME for tokens. Without the variable, the broker connects without security. TLS servers (kafka-secure, amqps, mqtts, secure-mqtt) verify the server with the system’s CA certificates. The README lists the variables of each broker and scheme.

Request/reply

  • A send operation with a reply channel: its function publishes the request and returns the reply, waiting at most timeout seconds. A subscriber of the reply channel hands replies to the waiting requests by correlation id: the message’s correlationId (a header, which the function fills in when the request has none, or a payload property), or FastStream’s correlation id when the document declares none. FastStream’s broker.request() isn’t used: Kafka doesn’t support it outside the test client, and RabbitMQ’s waits on amq.rabbitmq.reply-to instead of the reply channel of the document.
  • A send operation with a reply address in a header ($message.header#/replyTo): its function takes a reply_to argument and sends it in that header.
  • A receive operation with a reply channel: the handler returns the reply, which is published to the reply channel with the request’s correlation id header.
  • A receive operation with a reply address in a header: the handler returns the reply, which is published to the address in the request’s header.

Output

pyproject.toml          faststream[cli,…], pydantic; `pip install -e .`
README.md               how to run, the environment variables, what isn't generated
<package>/app.py        the FastStream application
<package>/brokers.py    the brokers and AMQP exchanges
<package>/subscribers.py
<package>/publishers.py
<package>/handlers.py   the handlers you write
<package>/models.py     Pydantic models (shared with apigenie-generator client python)
<package>/security.py   the security objects of the brokers, with security schemes
<package>/avro.py       the Avro records, with Avro payloads (and _avro.py, fastavro)
<package>/_models.py, _runtime.py, __init__.py, py.typed

handlers.py is written only when it doesn’t exist: it holds your code. After regenerating, add the handlers of new operations to it; mypy reports the missing ones.

Not generated

Reported as info lines or warnings:

  • channels of WebSocket and HTTP servers, which FastStream has no broker for

  • messages with payloads in other non-JSON Schema formats, Avro messages the application sends (see “Avro payloads”), and operations left without messages

  • the security schemes FastStream has no security object for (apiKey, httpApiKey, http, scramSha on other protocols, …). symmetricEncryption and asymmetricEncryption encrypt the messages, not the connection: an info line says that the handlers and publishers must do it

  • AMQP message bindings the messages of an operation disagree on, and content encodings other than gzip and deflate

  • servers no generated operation uses; a channel with several servers is served by its first one (choose others with the servers option)

  • reply addresses in payloads, AMQP cc and bcc, MQTT topic parameters inside a topic level

The MQTT message binding’s payloadFormatIndicator and messageExpiryInterval aren’t applied: FastStream’s MQTT publishers don’t set them.

Extensions

Extension Effect
x-apigenie.codegen.name on a component schema: the name of its model class; on an operation: the name of its function (in snake_case)
x-enum-varnames, x-enumNames names of the members of an enum
x-apigenie.codegen.disabled true on a channel or an operation leaves it out (with the operations of a channel left out)

The extensions can live in an overlay for this target instead of in the document, e.g. faststream.overlay.yaml:

overlay: 1.0.0
info:
  title: FastStream names
  version: 1.0.0
actions:
  - target: $.operations.publishOrder
    update:
      x-apigenie:
        codegen:
          name: announce_order
  - target: $.components.schemas.OrderStatus
    update:
      x-enum-varnames: [CREATED, PAID, SHIPPED]
apigenie-generator server faststream asyncapi.yaml --overlay faststream.overlay.yaml -o out

Options

-c, –config

Path to the configuration file to use. See the “Configuration” section for more information.

–overlay OVERLAY

Path to an OpenAPI Overlay (1.0 to 1.2) applied to the input before generating, e.g. one setting the extensions of the “Extensions” section. Can be repeated: the overlays apply in order, after those listed under commands.generate.server_faststream.overlay in the configuration file (with paths relative to it). Warnings and errors about a value an overlay set point to the overlay. The overlays apply to the AsyncAPI document bundled with the files it references.

–strict

Makes an overlay action whose target selects nothing an error (exit status 2, nothing is generated) instead of a warning (OVERLAY:LINE:COL: action #N (TARGET) selects no nodes.).

–check

Writes nothing: compares what the generator would write with the output directory, prints the files that would be created or changed, and exits with 1 when there are any (since 0.12.0). See Check that generated code is up to date.

-o, –output

Directory where the package is generated.

–package-name

Name of the generated distribution; the Python package is its snake_case form. Defaults to the title of the API.

–package-version

Version of the generated package. Defaults to the version of the API.

Configuration

version: '1.0'
commands:
  generate:
    server_faststream:
      servers: [production-kafka, customs-broker]
      enum_style: enum
      field_prefix: field_
      class_overrides:
        ShipmentId: ShipmentKey
Option Description
servers Keys of the servers to generate brokers for; all Kafka, AMQP and MQTT servers by default
enum_style enum (default): class Status(str, Enum) for enums of component schemas; literal: Literal[...] everywhere
field_prefix Prefix of field names that aren’t valid identifiers or clash with Pydantic, e.g. field_json (default field_)
class_overrides Class names of component schemas, by schema name

Verification

The codegen conformance suite runs this target on the Carpet Express AsyncAPI fixture. Level 1 compares the output with its snapshots and checks each feature, Level 2 runs the language's type checker and linter, and Level 3 runs the generated code in contract tests against the fixture. All targets.

This target's support tier is Tier 2. It goes through all three levels before every release, like Tier 1. It doesn't aim to support every feature: a feature of the fixture can stay unsupported, with the reason its manifest gives.

ResultsLevel 1Level 2Level 3
Release 0.12.0✓ snapshots of 3.0; 50 supported, 0 not yet✓ pass (python 3.12.11 faststream 0.7.7 pydantic 2.13.5 mypy 2.4.0, ruff 0.16.10)✓ 10 of 10 operations pass (python 3.12.11 faststream 0.7.7 pydantic 2.13.5 mypy 2.4.0, ruff 0.16.10)
Development version✓ snapshots of 3.0; 50 supported, 0 not yetruns before a releaseruns before a release
Was this helpful?