Terraform Provider Development Experience

Developing your own Terraform provider might be required when you have internal services and those services plus your Terraform modules are used by many teams. In such cases, having your own provider and distributing it makes Terraform code cleaner and allows you to avoid glue shell scripts.

I developed a few providers for our internal services. Below I want to list points that were interesting for me, or things I missed initially.

SDK version

There are two SDK versions available to develop a provider: SDKv2 and plugin framework. The latter is newer and recommended by Terraform, but that doesn’t mean SDKv2 is deprecated; it is still widely used. Probably it will take years before everything is migrated to the new one. In my case I used only the plugin framework as recommended. I also needed to read other providers which use SDKv2, and there are no problems doing that. There are differences between them, but probably because of the same protocol it is easy to understand what is going on even with SDKv2.

Pay attention to how import works

If you start provider development using scaffolding app template you will see code like this for a resource:

func (r *ExampleResource) ImportState(ctx context.Context, req resource.ImportStateRequest, resp *resource.ImportStateResponse) {
	resource.ImportStatePassthroughID(ctx, path.Root("id"), req, resp)
}

I initially didn’t pay much attention, but what is written here is directly linked to how you implement your Read function. Basically it means when “import” is happening your Read function will be executed, and only the id field will be available. This impacts how you define your ID — you should be able to read a resource purely by ID. It sounds easy, but in some cases a resource could be available only in a nested way. For example, let’s say you have a resource at this path: /policy/<policyID>/rules/<ruleID>.

Initially you could select ruleID as the id for your PolicyRule resource, which brings a problem with import. For that reason you should probably make your ID in this format <policyID>/<ruleID>. Doing this you can fetch the rule purely by ID:

func (r *ExampleResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {
	var id types.String
	resp.Diagnostics.Append(req.State.GetAttribute(ctx, path.Root("id"), &id)...)

	parts := strings.Split(id.ValueString(), "/")
	policyID := parts[0]
	ruleID := parts[1]

    // Define your http req

	// If applicable, this is a great opportunity to initialize any necessary
	// provider client data and make a call using it.
	// httpResp, err := r.client.Do(httpReq)
	// if err != nil {
	//     resp.Diagnostics.AddError("Client Error", fmt.Sprintf("Unable to read example, got error: %s", err))
	//     return
	// }

	// Save updated data into Terraform state
	resp.Diagnostics.Append(resp.State.Set(ctx, &data)...)
}

This of course means import should be run like this:

terraform import <resource-address> <policyID>/<ruleID>

Confusing Unknown and Null

I should admit, initially it was confusing to understand when an attribute is Unknown and when it is Null. The explanation is simple: Null is a value, explicitly set, or an optional attribute that is not provided in the config.

Often “computed” attributes are “unknown” during plan, they become known during apply. This is where stringplanmodifier.UseStateForUnknown() appears; basically this modifier tells Terraform to use the value currently in state, often used for an ID field. If such modifier is not used, you get “known after apply” every time.

Logging approach

In Terraform, structured logging is used. In my case I needed to log not only from the provider, but from the client code as well. I also wanted to keep some isolation (in case I would need to move the client code to a separate repository). For that I implemented an interface:

type Logger interface {
    Debug(ctx context.Context, msg string)
    Info(ctx context.Context, msg string)
}

On the provider side, add a logger struct:

type ClientLogger struct {
}

func (c *ClientLogger) Debug(ctx context.Context, msg string) {
    tflog.Debug(ctx, msg)
}

func (c *ClientLogger) Info(ctx context.Context, msg string) {
    tflog.Info(ctx, msg)
}

In your client code, accept a logger as a parameter:

type ExampleClient struct{
    Logger Logger
}

func (c *ExampleClient) Execute(ctx context.Context) {
    // this log appears if TF_LOG=INFO
    c.Logger.Info(ctx, "Starting execution")
    // do something
    // add more debugging logs if needed, this log appears if TF_LOG=DEBUG
    c.Logger.Debug(ctx, "Debugging information")
}

In my case, logging as much information as possible in case of errors was very helpful. Especially during Terraform executions in pipelines.

Also check diagnostics docs.

Keeping client code in same repo as your provider

There is a recommendation to not keep your client code for the service in the same repository as your provider. But for me it was a better option to avoid additional maintenance, as the service client wasn’t going to be used by anyone else. Probably it means if your company is big enough to have internal services, but not big enough to have Go SDKs for them, don’t hesitate to keep your service’s client code in the same repository. Of course keep isolation between client code and provider code (mixing them is definitely not a good idea).

For me it simplified overall development, as the service client and provider were implemented at the same time.