TODO these pages now have a lot of overlap with the tutorials. Work out what is duplicated and can be deleted, and how the remainder of the information should be structured.

Generate sources from IDL

While it is possible to use Mu by hand-writing your service definitions, message classes and clients in Scala, we recommend you use sbt-mu-srcgen to generate this code from Protobuf/Avro/OpenAPI IDL files.

IDL files are language-agnostic, more concise than Scala code, easily shared with 3rd parties, and supported by a lot of existing tools.

Mu can generate code from a number of different IDL formats:

• message classes, gRPC server and client from Protobuf .proto files (see the Protobuf section for detailed instructions)
• message classes, gRPC server and client from Avro .avpr or .avdl files (see the Avro section)
• message classes and REST client from OpenAPI .yaml files (see the OpenAPI section)

Plugin Installation

Add the following line to project/plugins.sbt:

addSbtPlugin("io.higherkindness" % "sbt-mu-srcgen" % "0.20.1")

How to use the plugin

The muSrcGen sbt task generates Scala source code from IDL files.

The plugin will automatically integrate the source generation into your compile process, so the sources are generated before compilation when you run the compile task.

You can also run the sbt task manually:

$ sbt muSrcGen

Import

You will need to add this import at the top of your build.sbt:

import higherkindness.mu.rpc.srcgen.Model._

Settings

muSrcGenIdlType

The most important sbt setting is muSrcGenIdlType, which tells the plugin what kind of IDL files (Avro/Protobuf/OpenAPI) to look for.

muSrcGenIdlType := IdlType.Proto // or IdlType.Avro or IdlType.OpenAPI

muSrcGenSerializationType

Another important setting is muSrcGenSerializationType, which specifies how messages should be encoded on the wire. This should match the format you chose for the muSrcGenIdlType setting:

• For Protobuf, choose SerializationType.Protobuf
• For Avro, choose either SerializationType.Avro or SerializationType.AvroWithSchema.
  • If you choose Avro, it means your client and server must always use exactly the same version of the schema.
  • If you choose AvroWithSchema, the writer schema will be included in every message sent, which introduces a bandwidth overhead but allows schema evolution. In other words, the server and client can use different versions of a schema, as long as they are compatible with each other. See the schema evolution section for more details on schema evolution.
• For OpenAPI, this setting is ignored, so you don’t need to set it.

muSrcGenSerializationType := SerializationType.Protobuf // or SerializationType.Avro or SerializationType.AvroWithSchema

Other basic settings

Note: The directories referenced in muSrcGenSourceDirs must exist. Target directories will be created upon generation.

Advanced settings

muSrcGenMarshallerImports

This setting specifies additional imports to add on top to the generated service files. This property can be used for importing extra codecs for your services.

By default:

• List(BigDecimalAvroMarshallers, JavaTimeDateAvroMarshallers) if muSrcGenSerializationType is Avro or AvroWithSchema and muSrcGenBigDecimal is ScalaBigDecimalGen
• List(BigDecimalTaggedAvroMarshallers, JavaTimeDateAvroMarshallers) if muSrcGenSerializationType is Avro or AvroWithSchema and muSrcGenBigDecimal is ScalaBigDecimalTaggedGen
• List(BigDecimalProtobufMarshallers, JavaTimeDateProtobufMarshallers) if muSrcGenSerializationType is Protobuf.

The JodaDateTimeAvroMarshallers and JodaDateTimeProtobufMarshallers are also available, but they need the dependency mu-rpc-marshallers-jodatime.

You can also specify custom imports with the following:

muSrcGenMarshallerImports := List(higherkindness.mu.rpc.srcgen.Model.CustomMarshallersImport("com.sample.marshallers._"))

See the Custom codecs section in core concepts for more information.

muSrcGenJarNames

You can use IDL files packaged into artifacts within your classpath, e.g. JAR files added to the classpath via libraryDependencies, or other sbt modules.

muSrcGenJarNames can be very useful when you want to distribute your IDL files without binary code (to prevent binary conflicts in clients).

The following example shows how to set up a dependency with another artifact or sbt module containing the IDL definitions (foo-domain):

//...
.settings(
  Seq(
      muSrcGenIdlType := IdlType.Avro,
      muSrcGenSerializationType := SerializationType.AvroWithSchema,
      muSrcGenJarNames := Seq("foo-domain"),
      muSrcGenTargetDir := (Compile / sourceManaged).value / "compiled_avro",
      libraryDependencies ++= Seq(
        "io.higherkindness" %% "mu-rpc-channel" % V.muRPC
      )
  )
)
//...

Implementation note: two-stage code generation

For gRPC services generated from an Avro or Protobuf definition, there are actually two stages of code generation at work.

1. The sbt-mu-srcgen plugin will parse your IDL files and transform them into Scala code. It writes this code to .scala files under target/scala-2.12/src_managed.

2. The generated Scala code contains @service macro annotations. When these files are compiled, the compiler will expand these annotations by executing a macro, which generates a load of boilerplate code to help with building a gRPC server or client.

For example, the following .proto file:

syntax = "proto3";

package foo.bar;

message MyRequest {
  string a = 1;
}

message MyResponse {
  string a= 1;
}

service MyService {
  rpc MyEndpoint (MyRequest) returns (MyResponse);
}

would result in a .scala file that looks like (slightly simplified):

package foo.bar

object myproto {
  final case class MyRequest(a: String)
  final case class MyResponse(a: String)

  @service(Protobuf) trait MyService[F[_]] {
    def MyEndpoint(req: MyReqeust): F[MyResponse]
  }
}

After the @service annotation is expanded at compile time, the entire generated code would look something like:

package foo.bar

object myproto {
  final case class MyRequest(a: String)
  final case class MyResponse(a: String)

  @service(Protobuf) trait MyService[F[_]] {
    def MyEndpoint(req: MyReqeust): F[MyResponse]
  }

  object MyService {

    def bindService[F[_]: ConcurrentEffect](
      implicit algebra: MyService[F]
    ): F[io.grpc.ServerServiceDefinition] = ...

    def client[F[_]: ConcurrentEffect: ContextShift](
      channelFor: higherkindness.mu.rpc.ChannelFor,
      channelConfigList: List[higherkindness.mu.rpc.channel.ManagedChannelConfig] = List(UsePlaintext),
      options: io.grpc.CallOptions = io.grpc.CallOptions.DEFAULT
    ): Resource[F, MyService[F]] = ...

    def clientFromChannel[F[_]: ConcurrentEffect: ContextShift](
      channel: F[io.grpc.ManagedChannel],
      options: io.grpc.CallOptions = io.grpc.CallOptions.DEFAULT
    ): Resource[F, MyService[F]]

    def unsafeClient[F[_]: ConcurrentEffect: ContextShift](
      channelFor: higherkindness.mu.rpc.ChannelFor,
      channelConfigList: List[higherkindness.mu.rpc.channel.ManagedChannelConfig] = List(UsePlaintext),
      options: io.grpc.CallOptions = io.grpc.CallOptions.DEFAULT
    ): MyService[F] = ...

    def unsafeClientFromChannel[F[_]: ConcurrentEffect: ContextShift](
      channel: io.grpc.ManagedChannel,
      options: io.grpc.CallOptions = io.grpc.CallOptions.DEFAULT
    ): MyService[F]

  }

}

You can see that the macro has generated a MyService companion object containing a number of helper methods for building gRPC servers and clients.

We will make use of these helper methods when we wire everything together to build a working server and client in the patterns section.