apigenie-generator server phoenix (AsyncAPI)

Tier 2 · Release 0.12.0: Level 1 ✓ · Level 2 ✓ · Level 3 ✓ · Verification

Generate a Phoenix application of the WebSocket and Kafka channels of an AsyncAPI document.

Usage

apigenie-generator server phoenix [...OPTIONS] INPUT_FILE

Description

apigenie-generator server phoenix generates from OpenAPI documents (see apigenie-generator server phoenix) and from AsyncAPI 3.x documents, telling them apart by their openapi or asyncapi field. Documents referencing other files are bundled first.

From an AsyncAPI document it generates a Mix project (Elixir 1.18 or later): the WebSocket channels on Phoenix 1.7 with Bandit, and the Kafka channels on brod and Broadway. By default the application is the one the document describes: it sends the messages of send operations and receives the messages of receive operations. Set the perspective option to counterpart to swap them; the counterpart has the Kafka channels only, since Phoenix serves WebSocket channels as the application. The project depends on snappyer, a NIF that needs a C++ compiler, so that brod reads Snappy-compressed records (the default of Go’s franz-go producers); LZ4 and zstd records need lz4b or ezstd added by hand.

The generator writes the channels of servers with the ws/wss and kafka/kafka-secure protocols, and channels with a WebSocket or Kafka binding in documents without servers. Channels of other protocols (MQTT, AMQP, HTTP) are skipped and reported as info.

defmodule MyApp.PickupDesk do
  @behaviour CarpetExpressEventsWeb.Channels.PickupDesk.Handler

  alias CarpetExpressEvents.Models

  @impl true
  def request_pickup(%Models.PickupRequest{} = request, _session) do
    # the reply gets the request's requestId when it has none
    {:reply, %Models.PickupConfirmed{request_id: nil, pickup: book(request)}}
  end
end

defmodule MyApp.LabelFarm do
  @behaviour CarpetExpressEvents.Kafka.RequestLabelRender

  @impl true
  def render_label(message, _metadata) do
    # sent to carpetexpress.label-render.results with the request's correlation id
    {:reply, %CarpetExpressEvents.Messages.LabelRendered{key: message.key, payload: render(message.payload)}}
  end
end
# config/config.exs
config :carpet_express_events, CarpetExpressEventsWeb.Channels.PickupDesk.Handler,
  impl: MyApp.PickupDesk

The project, for the application carpet_express_events (named by app, --package-name or the document’s title):

File Contents
lib/<app>/models/*.ex the structs of the schemas the payloads use, decoded and encoded by decode.ex and encode.ex (as the OpenAPI models of server phoenix)
lib/<app>/messages/*.ex per Kafka message, a struct (payload, headers, key) and the description of its record
lib/<app>/avro/*.ex the structs of the Avro records and the enums of the payloads, with their schemas
lib/<app>/kafka/security.ex the security schemes of the Kafka servers and brod’s client configuration of each
lib/<app>/kafka.ex the supervisor of the Kafka processes and a function per topic
lib/<app>/kafka/<operation>.ex per sent operation, its send (and request) functions; per received operation, the behaviour of its callbacks
lib/<app>/kafka/<operation>_consumer.ex per received operation, its Broadway pipeline
lib/<app>_web/user_socket.ex, channels/*.ex the socket of the Phoenix clients, a Phoenix channel per WebSocket channel and the behaviour of its callbacks (channels/<channel>/handler.ex)
lib/<app>_web/security.ex the behaviour checking the credentials of the WebSocket channels with security
lib/<app>_web/endpoint.ex the endpoint: the socket, then the plain WebSockets of the channels
lib/<app>/runtime/, lib/<app>_web/runtime/ the runtime of the Kafka processes and of the channels
config/*.exs, mix.exs, .formatter.exs, .credo.exs the project; config/runtime.exs reads PORT, KAFKA_BROKERS, KAFKA_SECURITY and, when Avro payloads are sent, KAFKA_SCHEMA_REGISTRY_URL
  • WebSocket channels, two transports. Each channel is a Phoenix channel: Phoenix clients join its topic on /socket/websocket, the address with its parameters as segments (/tracking/{shipmentId} is tracking:<shipmentId>), with the ws binding’s query as the join payload. The same channel is served as a plain WebSocket at its address, where the frames are the messages as the AsyncAPI document describes them (the prefix of <Root>Web.Runtime.PlainSocketPlug mounts the addresses under a path). Both transports decode the address parameters and the query, call the optional join/2 callback, decode received messages into their structs before the callbacks get them, and send what the channel module’s push_<message>/2 (one connection) and broadcast_<message> (every connection to the topic) functions send. A request’s callback returns {:reply, message}: the reply of the Phoenix protocol, or a frame on a plain WebSocket, with the request’s correlation id copied when it has none.
  • Received Kafka operations. A Broadway pipeline per operation reads its topic in its consumer group and hands each record to a callback of the operation’s behaviour (CarpetExpressEvents.Kafka.PublishShipmentEvents), named after the message. Several messages of a channel are told apart by a header or a payload property with a distinct constant value (ce_type), otherwise by the first message whose payload and headers decode. Records matching no message go to the optional on_unknown/1, records that don’t decode to the optional on_invalid/2 (and fail). Batching: batch_size and batch_timeout from x-apigenie-broadway of the operation’s (or channel’s) Kafka binding, else the broadway option, else Broadway’s 100 records and 1000 ms; the pipeline module’s configuration overrides them, and the consumer group and offset reset policy too.
  • Sent Kafka operations. send_<message>/1 checks the headers of a message, encodes it and returns once the brokers have the record, keyed by key.
  • Request/reply on Kafka, with a reply channel: request_<message>/2 sends the request with its correlation id (generated when it has none) and waits for the reply (:timeout, 30 seconds by default), read by a consumer of the reply topic without consumer group; a replier’s callback returns the reply, sent to the reply topic with the request’s correlation id.

Group ids come from the groupId schemas of the Kafka operation bindings (their const, single enum value, default or first example), else <app in kebab case>-<operation>. Client ids come from their clientId schemas the same way: the brod client of the producers has the application’s, <app in kebab case> by default, and the Broadway pipelines and the consumers of replies the operation’s, else <app in kebab case>-<operation>. The partitions, replicas and topic configuration of channel bindings are documented on the topic functions; the generated code creates no topics.

Not supported, and reported as warnings: payloads of non-JSON schema formats other than Avro (the messages are skipped, and operations left without a message too), reply addresses given by runtime expressions, Kafka topics with parameters, WebSocket addresses whose segments mix text and parameters, and security schemes a WebSocket handshake can’t carry (userPassword, X509, the SASL schemes). Channels of other protocols (MQTT, AMQP, HTTP) are out of scope. The models check the schemas’ constraints, if/then/else, dependentRequired, dependentSchemas and not when they decode a payload.

Kafka security

Each SASL or client certificate scheme of the Kafka servers has a function of <Root>.Kafka.Security, named after the scheme key, returning brod’s client configuration of the scheme: kafka_scram(username, password) for the SASL scheme kafkaScram (over TLS on kafka-secure servers), kafka_mtls(cert_file, key_file, ca_file) for the client certificate of kafkaMtls. client_config(scheme) takes the credentials from environment variables named after the scheme key, e.g. KAFKA_SCRAM_USERNAME and KAFKA_SCRAM_PASSWORD, or KAFKA_MTLS_CERT, KAFKA_MTLS_KEY and KAFKA_MTLS_CA. The Kafka supervisor connects with the scheme of its :security option, which config/runtime.exs reads from KAFKA_SECURITY; :client_config adds to brod’s configuration.

config :carpet_express_events, :kafka, brokers: "kafka:9093", security: "kafkaScram"

WebSocket security

A channel’s security is the schemes of its operations, else of its servers, that a WebSocket handshake carries: API keys in the query, a header or a cookie (httpApiKey), basic credentials and bearer tokens (http with the basic or bearer scheme, oauth2, openIdConnect). A connection to a channel with security is accepted when the implementation of <Root>Web.Security accepts its credential of one of the schemes, and refused otherwise: the join of a Phoenix client with {:error, %{reason: "unauthorized"}}, the handshake of a plain WebSocket with 401 Unauthorized. authenticate/2 returns {:ok, principal} to accept the connection, and principal becomes the session’s principal; it returns {:error, reason} to refuse it.

defmodule MyApp.Security do
  @behaviour CarpetExpressEventsWeb.Security

  # each returns {:ok, merchant} or {:error, :invalid}
  @impl true
  def authenticate("streamToken", {:api_key, token}), do: MyApp.Tokens.verify(token)
  def authenticate("oauth2", {:bearer, token, scopes}), do: MyApp.OAuth.verify(token, scopes)
end
config :carpet_express_events, CarpetExpressEventsWeb.Security, impl: MyApp.Security

Without an implementation, connections to channels with security are refused. The headers schema of a channel’s ws binding is decoded into session.params.headers. Phoenix clients send query parameters as the parameters of the socket or of the join, and only x- headers reach Phoenix channels (the endpoint’s connect_info: [:x_headers]); plain WebSockets get every header.

Avro payloads

A message whose payload is an Avro schema (a schemaFormat of application/vnd.apache.avro) carries a struct of its record, lib/<app>/avro/<record>.ex, with its schema, its fields and their defaults; Avro enums are atoms. The record is written and read by <Root>.Runtime.Kafka.AvroCodec, a dependency-free encoder and decoder of the Avro binary encoding with the logical types date, time-*, timestamp-* and local-timestamp-* (as Date, Time, DateTime and NaiveDateTime).

The id of the record’s schema goes where the Kafka binding’s schemaIdLocation says: in the apicurio.value.globalId header, or in the payload after a zero byte (the Confluent wire format). The sent schemas are registered in a Confluent-compatible registry, the schemaRegistryUrl of the Kafka server by default, under the subject of the binding’s schemaLookupStrategy (<topic>-value by default), the first time each is sent; the :schema_registry option of the Kafka supervisor (url, username, password, or a register function replacing the requests) and KAFKA_SCHEMA_REGISTRY_URL change it. The registry is not asked when reading: the reader’s schema is the generated one. Unions are supported when they make a type nullable (["null", T]); messages with other unions, with decimals or whose payload is not a record are skipped with a warning, as are request/reply operations with Avro messages.

Options

-c, –config

Path to the configuration file to use. See the “Configuration” section for more information.

–check

Writes nothing: compares what the generator would write with the output directory, prints the files that would be created or changed, and exits with 1 when there are any (since 0.12.0). See Check that generated code is up to date.

-o, –output

Directory where the project is generated.

–package-name

Name of the Mix application, converted to snake_case (carpet-express-events gives carpet_express_events, root module CarpetExpressEvents); defaults to the title of the API (events when it gives no name). Its kebab case is the base of default group ids.

–package-version

Version of the Mix project; 0.1.0 by default.

-p, –property

Applies to OpenAPI documents only; using it with an AsyncAPI document is an error.

Configuration

The options are read from the asyncapi section of the configuration of apigenie-generator server phoenix; its other keys configure the generation from OpenAPI documents.

version: '1.0'
commands:
  generate:
    server_phoenix:
      asyncapi:
        app: carpet_express_events
        namespace: CarpetExpressEvents
        perspective: application
        broadway: { batch_size: 100, batch_timeout: 1000, concurrency: 10 }
Option Type Default Description
app string from --package-name or the title Name of the Mix application (snake_case)
namespace string the application in PascalCase Root module of the generated modules; the web modules are in <Root>Web
perspective application | counterpart application The described application, or its counterpart, receiving what it sends
field_prefix string field_ Prefix of struct fields whose property names give no valid name
module_overrides map Module names of payload schemas, e.g. ShipmentId: ShipmentRef
dev_deps boolean true credo and dialyxir as dependencies of the dev and test environments
broadway map batch_size, batch_timeout (ms) and concurrency of the pipelines

Verification

The codegen conformance suite runs this target on the Carpet Express AsyncAPI fixture. Level 1 compares the output with its snapshots and checks each feature, Level 2 runs the language's type checker and linter, and Level 3 runs the generated code in contract tests against the fixture. All targets.

This target's support tier is Tier 2. It goes through all three levels before every release, like Tier 1. It doesn't aim to support every feature: a feature of the fixture can stay unsupported, with the reason its manifest gives.

ResultsLevel 1Level 2Level 3
Release 0.12.0, server phoenix✓ snapshots of 3.0; 41 supported, 0 not yet✓ pass (Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]; Mix 1.20.4 (compiled with Erlang/OTP 29))✓ 11 of 11 operations pass (Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]; Mix 1.20.4 (compiled with Erlang/OTP 29))
Release 0.12.0, server phoenix (counterpart)✓ snapshots of 3.0; 30 supported, 0 not yet✓ pass (Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]; Mix 1.20.4 (compiled with Erlang/OTP 29))✓ 11 of 11 operations pass (Erlang/OTP 29 [erts-17.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]; Mix 1.20.4 (compiled with Erlang/OTP 29))
Development version, server phoenix✓ snapshots of 3.0; 41 supported, 0 not yetruns before a releaseruns before a release
Development version, server phoenix (counterpart)✓ snapshots of 3.0; 30 supported, 0 not yetruns before a releaseruns before a release
Was this helpful?