Printing Exception Tracebacks from Pulumi Dynamic Resources

The Problem

At x-RD, we make extensive use of Pulumi Dynamic Providers to bridge gaps in Pulumi Providers and to create special glue between various systems and services. As such, it's important that when an error occurs, we're able to quickly understand the exact nature of the failure and get in to implementing a fix quickly.

When an exception occurs in a Dynamic Resource, Pulumi will print the exception and exception message, but won't print the traceback. This is really annoying, because it makes it significantly harder to diagnose the problem that's occurring.

Problem Example

Perhaps we're indexing into a list that we've dynamically retrieved from an API (say with the boto3 library), and the API has returned an empty list because of some error. So we tried to simulate that with a __main__.py that intentionally hits an "index out of range" exception.

And... there was no traceback!

Hypothesis

We know that we can print messages (with pulumi.error()) from inside the dynamic resources, and that a raised exception will stop execution and will be printed by Pulumi.

Therefore, we should be able to decorate or wrap each of the entry methods to our Dynamic Resources such that we'll catch any exceptions that occur, nicely format and print the traceback, and then re-raise the exception so that Pulumi handles it as it originally would, but provides us useful output.

Creating the Decorator

The wrapper we actually want to put around each of the methods is pretty simple - catch everything, print the traceback, and then raise again.

Attempt 1 - Decorators Class Decorators

Code example (Pulumi Project) for this section can be found in the 1-class-decorator folder.
First idea - what if we decorate the class, such that all methods are wrapped and replaced at instantiation time? Let's implement that with a class decorator that patches the methods we care about... and it gave us the information we wanted! We're done here right?

We could be, but we ended up taking a different approach. This solution might work well enough for your use case, but class decorators don't work (for us).

In our codebase we define some base Dynamic Resource Provider classes with some shared setup and utility functionality. I would hazard a guess that many other people implementing Dynamic Resources against a single service would end up doing something similar. So what happens when we subclass ExampleProvider to provide a new resource ExampleResource2?

If we add a subclass and new resource, we do get a traceback for ExampleResource but we don't get a traceback for ExampleResource2.

We could easily get around this by requiring that all dynamic provider classes be decorated. But we're already inheriting from a common base class, so let's see if we can find a solution that Just WorksTM.

Attempt 2 - Wrapping in Dynamic Provider __init__
We know our decorator works. What if instead we patch it in __init__? Our subclass dynamic providers don't override __init__ so it'll pass through, and if they really need to override the behaviour then they can define their own __init__, as is the standard way with Python subclassing. Let's dynamically patch the methods there.

Now that we know we can catch and print the exception, let's test how what we've done affects the stack behavior in normal operation. I comment out the print(empty_list[1]) line so that our dynamic resource will create successfully, and run a stack up...

Perfect, it works. Now at x-RD we like our stack behaviour to be as clean as possible, in practice one of the requirements for that is that the diff after we bring a stack up should report no changes. So we test that that is the case with pulumi preview --expect-no-changes...

Alright it worked, that's a wrap (pun intended). We've done it.

Conclusion

We can catch, format, print, and re-raise an exception in a Dynamic Provider by wrapping the provider methods. This greatly improves our ability to rapidly develop and diagnose issues with our Dynamic Resources.

I hope that this might be helpful to anyone else out there also struggling with debugging Pulumi Dynamic Resources.