Following the link in the comment:

Turns out the session should be configured to have device count > 1:

config = tf.ConfigProto(device_count={"CPU": 8})
with tf.Session(config=config) as sess:
   ...

Kind of shocking that I missed something so basic, and no one could pinpoint to an error which seems too obvious.

Not sure if it's a problem with me or the TensorFlow code samples and documentation. Since it's Google, I'll have to say that it's me.