Update: I blogged about this. When researching the blog post, I found some potential for improvement; I've updated the relevant parts of this answer. For more detail on why I think this is better than all other suggestions here, or the default behavior, read the entire post :)

I have now gone with the following approach, which seems to work OK, even if not 100 % compliant with what I was looking for:

• Create a class PassthroughExceptionHandler:

  public class PassthroughExceptionHandler : IExceptionHandler
  {
      public Task HandleAsync(ExceptionHandlerContext context, CancellationToken cancellationToken)
      {
          // don't just throw the exception; that will ruin the stack trace
          var info = ExceptionDispatchInfo.Capture(context.Exception);
          info.Throw();
          return Task.CompletedTask;
      }
  }
  

• Let that class replace the IExceptionHandler service of Web API:

  config.Services.Replace(typeof(IExceptionHandler), new PassthroughExceptionHandler());
  

• Create a middleware class which does what I want:

  public class ExceptionHandlerMiddleware
  {
      public override async Task Invoke(IOwinContext context)
      {
          try
          {
              await Next?.Invoke(context);
          }
          catch (Exception ex)
          {
              // handle and/or log
          }
      }
  }
  

• Register that middleware first in the stack:

  app.Use<ExceptionHandlerMiddleware>()
     .UseStageMarker(PipelineStage.Authenticate)
     // other middlewares omitted for brevity
     .UseStageMarker(PipelineStage.PreHandlerExecute)
     .UseWebApi(config);
  

I will still award the bounty to anyone who comes up with (bounty expired...) I'm still looking for a better solution, which, for example, breaks when an unhandled exception is thrown. (This approach makes VS break when I rethrow the exception in the handler, but the original call stack is lost; I have to set a breakpoint at the faulting line and debug again to be able to intercept the state when an exception is thrown.)