mill-http4s-grpc

A Mill plugin that generates http4s-grpc service bindings from .proto files — the Mill counterpart of upstream's sbt-http4s-grpc — plus the gRPC runtime those bindings need, vendored so its dependency versions are ours to move.

Messages come from mill-protoc, not ScalaPB, as of 0.1.0. That is where protobuf editions support comes from, and it is why the generated messages are Scala 3 case classes with enum oneofs rather than ScalaPB's shapes.

Two published artifacts:

Module	Artifact	What it is
plugin	nscn.polaris2::mill-http4s-grpc-plugin_mill1	the Mill plugin + the http4s service generator
core	nscn.polaris2::mill-http4s-grpc-core	the gRPC runtime, package nscn.polaris2.grpc

Usage

Everything lives in the NorthstarCN Forgejo Maven registry, not Maven Central, so the repository has to be declared twice — the build header resolves the plugin itself, and the module resolves the runtimes that Http4sGrpcModule adds to mvnDeps. Omitting the second one fails at resolvedMvnDeps with core not found:

//| repositories:
//| - "https://forgejo.wolf109909.top/api/packages/NorthstarCN/maven"
//| mvnDeps:
//| - "nscn.polaris2::mill-http4s-grpc-plugin_mill1:0.1.1"

package build
import mill.*, scalalib.*
import nscn.polaris2.mill.Http4sGrpcModule

object app extends ScalaModule, Http4sGrpcModule {
  def scalaVersion = "3.9.0"
  def jvmVersion = "25"
  def scalacOptions = Task { super.scalacOptions() ++ Seq("-release:25") }

  def repositories = Seq("https://forgejo.wolf109909.top/api/packages/NorthstarCN/maven")
}

Put your .proto files in app/protobuf/ and run ./mill app.compile. The module adds nscn.polaris2::mill-http4s-grpc-core and nscn.polaris2::mill-protoc-runtime to mvnDeps for you, and wires both sets of generated sources into generatedSources.

Adding your own dependencies must go through super: def mvnDeps = Task { super.mvnDeps() ++ Seq(...) }. A bare def mvnDeps = Seq(...) compiles cleanly — Mill's build-file compiler inserts override for you — and silently drops both runtimes, which surfaces much later as unresolved nscn.polaris2.grpc.* symbols.

For a service Greeter declared in proto package hello.world inside hello.proto, you get (mill-protoc appends the file's base name as a package segment unless protocFlatPackage is on, so the Scala package is hello.world.hello):

trait Greeter[F[_]] {
  /** Say hello once. */
  def sayHello(request: HelloRequest, ctx: Headers): F[HelloReply]
}

object Greeter {
  def fromClient[F[_]: Concurrent](client: Client[F], baseUri: Uri): Greeter[F]
  def toRoutes[F[_]: Temporal](serviceImpl: Greeter[F]): HttpRoutes[F]
}

Comments on the service and on each rpc are carried into the generated Scaladoc.

Well-Known Types in RPC position

mill-protoc does not generate the Well-Known Types, it maps them, so an rpc that takes or returns one gets the mapped Scala type rather than a message class:

rpc Now(google.protobuf.Empty) returns (google.protobuf.Timestamp);

def now(request: Unit, ctx: Headers): F[java.time.Instant]

The full table is mill-protoc's — Empty is Unit, Timestamp is Instant, Duration is java.time.Duration, Any is ProtoAny, the wrappers are their primitives. Set (nscn.polaris2.protoc.file).wkt_mapping = RAW on the file declaring the service to get protobuf-java's own classes instead; the option is read from that file, not from google/protobuf/*.proto, and there is no per-rpc override because an rpc has nowhere to hang one.

The wire format is unaffected either way — the mapping applies on top of protobuf-java's serialization, through ProtoCodec.codecForWellKnown.

Fixed in 0.1.1. 0.1.0 emitted com.google.protobuf.empty.Empty here — the name GenContext.fullName mechanically derives from the proto package and file base name, for a type mill-protoc never generates — and the generated file did not compile.

Consumers must be on Scala 3.9.0 or newer, and on a JDK 25 runtime. core is built with 3.9.0, which emits TASTy 28.9 — a 3.3.x module cannot read it — and targets JDK 25 bytecode (class file 69). mill-protoc's runtime is the same on both counts.

Migrating from 0.0.x

0.1.0 replaces the message stack. Nothing about the gRPC side changed — the wire format, the trait shape and the four streaming combinators are the same — but every message type a consumer touches is now generated by a different generator.

0.0.x	0.1.x
com.thesamet.scalapb::scalapb-runtime	nscn.polaris2::mill-protoc-runtime
scalapb.GeneratedMessage	nscn.polaris2.protoc.runtime.GeneratedMessage
com.google.protobuf.any.Any	nscn.polaris2.protoc.runtime.ProtoAny
nscn.polaris2.grpc.codecs.ScalaPb	nscn.polaris2.grpc.codecs.ProtoCodec
http4sGrpcSources, http4sGrpcFlatPackage, http4sGrpcProtoc*, …	protocProtoSources, protocFlatPackage, protocPath, … (from ProtocModule)
http4sGrpcGenerateMessages = false	gone — see below
ScalaPB lenses, java_conversions, ascii_format_to_string	not implemented; mill-protoc rejects them rather than ignoring them
proto2 and proto3	proto2, proto3 and editions
a Well-Known Type in rpc position was a ScalaPB message	it is the mapped type, e.g. Unit for Empty — see above (0.1.1)

GrpcStatus.addDetails still takes any generated message and packs it; details is now a List[ProtoAny] instead of a List[com.google.protobuf.any.Any]. ProtoAny.unpackTo[A] checks the type URL and throws when it names something else, where ScalaPB's unpack would have parsed the bytes anyway.

http4sGrpcGenerateMessages is gone because the module no longer owns the message target: ProtocModule does, and it always runs. If the messages genuinely come from another module, depend on that module and do not mix in Http4sGrpcModule — you would be generating them twice.

How the generation works

Http4sGrpcModule extends mill-protoc's ProtocModule and contributes one extra target to the protoc invocation it already performs, through ProtocModule.protocExtraTargets(dest, options) (added in mill-protoc 0.0.2 for this):

• <dest>/ — the messages, from mill-protoc's scala3 generator
• <dest>/http4s-grpc/ — the service bindings, from this project's generator

One protoc run, one include path, one set of options. The subdirectory is not cosmetic: a service Greeter and a message Greeter in the same file would otherwise both want Greeter.scala. The options are passed through deliberately — flat_package and (nscn.polaris2.protoc.file).package_name decide where the message types land, and a binding that resolves a different name does not compile.

mill-protoc's own generator ignores service declarations entirely, so the two targets never produce the same file.

Code generation runs in-process in the Mill JVM rather than in a sandboxed classloader the way sbt-protoc does. sbt needs the sandbox because it ships an old protobuf-java on its own classpath; Mill ships neither protobuf-java nor a code generator, so there is nothing to conflict with. A SandboxedJvmGenerator handed to protocExtraTargets still gets a properly isolated classloader from ProtocModule.

The core runtime

core/ is the core module of http4s-grpc, repackaged from org.http4s.grpc to nscn.polaris2.grpc. It exists so dependency versions are controlled here rather than pinned by whatever org.http4s::http4s-grpc:0.3.0 was published against. Bump them in one place — object Versions in build.mill:

val cats = "2.13.0"
val catsEffect = "3.7.1"
val fs2 = "3.14.0"
val http4s = "1.0.0-M48"
val protoc = "0.0.2"   // mill-protoc: runtime, codegen, bridge and the Mill plugin
// ...

The ported sources track upstream closely but are not held byte-identical to it. Re-porting a newer upstream starts mechanically:

# from an http4s-grpc checkout, for core/src/main/scala/org/http4s/grpc/**.scala.
# Sources only - the tests under core/test/ are ours, see below:
sed -e 's|^package org\.http4s\.grpc|package nscn.polaris2.grpc|' \
    -e 's|^import org\.http4s\.grpc\.|import nscn.polaris2.grpc.|' \
    <upstream-file> > core/src/nscn/polaris2/grpc/<file>

Then review the result rather than taking it as-is:

• Re-apply the mill-protoc port. Upstream is ScalaPB-based, so a fresh copy of codecs/ScalaPb.scala, GrpcStatus.scala or GrpcStatusDetails.scala will import scalapb.* and com.google.protobuf.any.Any. The replacements are codecs/ProtoCodec.scala, nscn.polaris2.protoc.runtime.GeneratedMessage and ProtoAny; ProtoAny carries no wire format of its own, so GrpcStatusDetails converts through toJava to serialize and reads details with ProtoAny.fromJava(JavaAny.parseFrom(input.readByteArray())).
• Keep codecs/Http4sInternals.scala (it is ours, not upstream's) and re-apply the NamedHeaders.scala redirect described below.
• Re-apply the GrpcStatusDetails.fromByteVector unknown-field fix described below.
• Modernise syntax that upstream only carries to stay 2.13-cross-compilable. Upstream uses the x: _* vararg splice, which Scala 3.4+ deprecates; this port spells those x* instead (four sites in ClientGrpc.scala, one in GrpcStatus.scala).
• Do not copy upstream's core tests over ours. Upstream's entire core suite is a single assertEquals(ExitCode.Success, ExitCode.Success) placeholder; its real tests live in codegen/testing, which needs generated code and so has no counterpart here. core/test/ is original work for this project and is the safety net for a re-port.
• Check upstream's build.sbt for dependency changes, then run ./mill core.test.

The deviations are deliberate, so core compiles warning-free with no -Wconf suppressions. Prefer fixing a new warning over silencing it.

The biggest deviation: http4s internals

Upstream codecs/NamedHeaders.scala uses three private[http4s] APIs — ParseResult.fromParser, parser.AdditionalRules.NonNegativeLong and internal.parsing.CommonRules.ows. It may do so only because it declares itself in package org.http4s.grpc. Repackaging puts them out of reach, so they are reimplemented on public cats-parse APIs in codecs/Http4sInternals.scala.

That is a net win: those internals carry no compatibility guarantee, so depending on them made every http4s upgrade a gamble. Concretely, NamedHeaders.scala differs from upstream by dropping the two internal imports and calling:

Http4sInternals.nonNegativeLong <* Http4sInternals.ows   // was AdditionalRules.NonNegativeLong <* ows
Http4sInternals.fromParser(parser, "...")(s)             // was ParseResult.fromParser(...)

When re-porting, grep the result for AdditionalRules, CommonRules and ParseResult.fromParser — any hit means the redirect needs re-applying, or that upstream has started using a different internal API.

ClientGrpc.scala also uses org.http4s.h2.H2Keys, but that one is public.

The other deviation: unknown fields in grpc-status-details-bin

GrpcStatusDetails.fromByteVector hand-decodes google.rpc.Status off a CodedInputStream. Upstream's unknown-field branch reads:

case _ => () // ignore unknown fields

That consumes the tag but not the field's payload, so the payload bytes are then read as the next tag. One unknown field does not get ignored — it derails the rest of the message, and since the whole decode is wrapped in catch NonFatal => None, the failure is silent: the entire grpc-status-details-bin header is dropped and the caller sees a status with no details. This port skips the field properly:

case tag => if (!input.skipField(tag)) done = true

It only bites against a peer that puts something beyond fields 1–3 in google.rpc.Status, which is why upstream has not noticed, but the failure mode is silent data loss rather than an error. GrpcStatusSuite covers trailing and leading unknown fields, varint and length-delimited. Worth sending upstream.

The service generator

plugin/src/nscn/polaris2/mill/generator/ is a port of upstream's Http4sGrpcCodeGenerator / Http4sGrpcServicePrinter, rebased from ScalaPB's DescriptorImplicits onto mill-protoc's GenContext. Every type name it emits comes from ctx.fullName, which is the same call mill-protoc's own message printers make — so the two targets cannot disagree about where a message landed.

It is vendored rather than resolved: the upstream generator carries a downstream prefixRoot fix that is in no published artifact, and org.http4s:http4s-grpc-generator is published for Scala 2.12 only, so a Scala 3 Mill plugin could not depend on it in any case. It stays a plain ProtocCodeGenerator, so it also runs as a standalone protoc plugin via its inherited main.

Two things the port had to supply itself, both previously borrowed from the ScalaPB compiler:

• Qualification. Upstream prepends _root_. unconditionally, which yields _root_._root_.foo.Bar on a name ScalaPB had already qualified — and ScalaPB did exactly that whenever the top-level Scala package was called build, which is where every Mill build file lives. GenContext.fullName qualifies once, always, so the question does not arise for type references; prefixRoot remains for the one name the printer assembles itself (the trait's own), and is idempotent. example/app/protobuf/build_pkg.proto keeps the case covered.
• Scaladoc. ProtobufGenerator.asScalaDocBlock went away with ScalaPB. Http4sGrpcServicePrinter.scalaDocBlock replaces it, and the comments themselves are read out of SourceCodeInfo by path — [6, serviceIndex] for a service, [6, serviceIndex, 2, methodIndex] for a method — because protoc reports comments as paths into the FileDescriptorProto rather than hanging them off the descriptors.

Configuration

Most of it is ProtocModule's, and documented in mill-protoc's README:

Task	Default	Purpose
protocProtoSources	<module>/protobuf	where .proto files live
protocVersion	what mill-protoc was built against	which protoc to resolve
protocPath	None	use your own protoc binary instead of resolving one
protocFlatPackage	false	drop the per-file package segment
protocRetainSourceCodeInfo	false	keep comments in the embedded descriptors
protocOptions	empty	options passed to the generators verbatim
protocArgs	empty	extra protoc arguments
protocExtraTargets(dest, options)	this module's service target	additional generators
http4sGrpcVersion	0.1.0	nscn.polaris2::mill-http4s-grpc-core runtime version

protoc is resolved from Maven Central as a platform-specific binary and cached by Mill, rather than extracted at runtime — the latter races when several modules generate in parallel. Setting protocPath skips that resolution entirely rather than downloading a binary it then ignores, which matters for offline builds.

Protos importing a Well-Known Type (google/protobuf/timestamp.proto and friends) need no configuration: the protoc artifact ships no include/ directory, so ProtocModule unpacks the .proto files out of the resolved dependencies — protobuf-java carries the Well-Known Types, and it arrives transitively with the runtime. This is the one place 0.1.0 is less fiddly than 0.0.x, where the same case needed http4sGrpcSearchDeps = true.

Layout

core/src/nscn/polaris2/grpc/          gRPC runtime, ported from http4s-grpc
  codecs/ProtoCodec.scala             scodec codecs over mill-protoc messages
core/test/protobuf/echo.proto         the generated message the suites send
core/test/src/nscn/polaris2/grpc/     ours, not upstream's (see the re-port notes)
  GrpcRoundTripSuite.scala            ClientGrpc <-> ServerGrpc, all four shapes
  GrpcStatusSuite.scala               status/details model + google.rpc.Status wire format
  NamedHeadersSuite.scala             the grpc-* header codecs
plugin/src/nscn/polaris2/mill/
  Http4sGrpcModule.scala              the Mill module trait, extending ProtocModule
  generator/
    Http4sGrpcCodeGenerator.scala     protoc plugin entry point
    Http4sGrpcServicePrinter.scala    emits the service trait/client/routes
plugin/test/src/...                   prefixRoot and Scaladoc rendering
example/
  app/protobuf/hello.proto            all four streaming shapes
  app/protobuf/build_pkg.proto        the `build` package regression case
  app/src/Main.scala                  implements the generated trait, both directions

core/test mixes in ProtocModule and generates Echo from echo.proto, so the codecs are exercised against genuinely generated code rather than a hand-written stand-in that would be free to drift. Up to 0.0.2 that role was played by ScalaPB's com.google.protobuf.any.Any, which happened to be a generated message; mill-protoc's ProtoAny deliberately is not.

Running the example

./mill '{core,plugin}.test'
./mill '{core,plugin}.publishLocal'
cd example && ./mill app.compile

Use the brace form, not ./mill core.test plugin.test. test and publishLocal are Mill commands, so a second selector on the same line is parsed as an argument to the first one, not as a second target. ./mill core.test plugin.test passes plugin.test to munit as a test-name filter: the plugin suite never runs, munit reports a spurious failure, and the build still exits 0. ./mill __.test works too.

The example resolves this project's artifacts from ~/.ivy2/local, so publishLocal must run first and must be re-run after any change. It is the only check that the plugin actually loads and that both protoc targets agree; a green plugin.compile proves neither.

Maintaining: the two Scala versions are pinned, and for different reasons

core and plugin deliberately do not share a Scala version.

plugin is pinned to 3.8.2 by a hard two-sided constraint — both directions were hit while building this:

• Too new (e.g. 3.9.0): the plugin emits TASTy 28.9. Mill 1.1.9 compiles build.mill with a compiler that accepts 28.0–28.8, so it refuses to load the plugin: "Forward incompatible TASTy file has version 28.9 ... expected stable TASTy from 28.0 to 28.8". The plugin compiles and publishes fine; it is simply unusable.
• Too old (e.g. 3.7.4): mill-libs pulls org.scala-lang:scala-library:3.8.2, whose TASTy is 28.8, and a 3.7.4 compiler reads only up to 28.7 — this module then fails to compile against its own dependencies.

It must also match what mill-protoc builds mill-protoc, mill-protoc-codegen and mill-protoc-bridge with, since this plugin links against all three. Bump it only together with Versions.mill, and verify by actually running the example.

core is at 3.9.0 and is free to move, because the plugin never links against it; it only emits a dependency coordinate. The constraint here points at consumers instead: raising core's Scala version raises the minimum for everyone who depends on it.

Both modules pin jvmVersion = "25", but they emit different bytecode. core also sets -release:25, so it emits class file 69 and requires a JDK 25 runtime — that is a deliberate choice to match the JDK this project targets. plugin leaves Mill's default -release:17 alone and emits class file 61, because a Mill plugin has to be loadable by whatever JDK the consumer runs Mill on. Do not "unify" these.

Licensing

This project is MIT licensed — see LICENSE.

core and the service generator are derived from http4s-grpc, itself derived from fs2-grpc; both are MIT. The original copyright headers are retained on those files, and the upstream copyright notices are reproduced under Third-party code in LICENSE. mill-protoc, which supplies the message stack, is Apache-2.0 and is depended on rather than vendored.