zBrush Subdiv Normal Map Workflow

Working with subdivided low poly models in zBrush is great and all, but getting a good normal map out of it can be tricky. If you haven't already watched the super fantastic Handplane 3d videos, you probably should (they explain normal maps, hard edges/smoothing splits, UVs, and tangent space). The main things are: different applications calculate tangent-space normals differently, and the smoothing of a low poly mesh needs to work with the tangent-space normal map.

So, object-space and tangent-space normal maps are different (the Polycount wiki has a nice rundown), and it is possible to quickly and easily create an object-space normal map in zBrush, and then use xNormal and this Unity tangent basis plugin to convert it to a tangent-space normal map that will work properly in Unity (or to xNormal's tangent space or anything else you can find a plugin for. Handplane is also great, but I've been running into Unity export glitches).

These images are of the same same model running in Unity, the 'zBrush' mesh is using tangent-space normals straight out of zBrush, the 'xNormal' and 'Unity' meshes are using zBrush object-space normals which were converted to tangent space with and without the Unity tangent basis calculator plugin. 'Unity' and 'xNormal' look mostly similar (I think hard-surfaces tend to have more issues), but there are differences where the tail meets the body and at the tip of the ear. The zBrush map obviously has a lot of problems, though. The bottoms of the hooves and the edge of the mane benefit from smoothing/UV splits for the 'xNormal' and 'Unity' maps, but the zBrush map uses 'Smooth low res normals', which means that the entire mesh needs to be smoothed (this might have been fixed  zBrush 4r7, though?). Beyond this, there are obviously huge shading problems, especially around the eyes, nostrils and tail. 

 

So, to get the good normal maps:

Create an object-space normal map in zBrush (Enable 'Flip G' and 'Flip B'; zBrush seems to invert the y and z axises, which is consistent with Deformation/Translate being backwards for those but not x, I guess?). 'Adaptive' only seems to add glitches, so I disable that, and (because object-space maps are absolute and not relative) smooth low poly normals shouldn't make a difference either way.

Next; either export the low poly mesh from zBrush and fix the smoothing in Maya/3ds/whatever, or use the model that you originally imported in to zBrush (the exported low poly might match the shape a bit better).

Now, in xNormal (if you're using the Unity Tangent Basis calculator, make sure it is enabled under plugins), go to Tools, Object/Tangent space converter, add your low poly mesh, the zBrush object-space normal map, add an output path, confirm the thing if you're creating a new file, and hit 'Generate'.

 

So there you go; no cages, no projection errors, no super slow 'Match UV' baking, just perfect shading forever.

I added a PBR version of this mesh to sketchfab (the one below is unlit/doesn't have a normal map) if you wanna check out a converted normal map in action or look at the wires or whatever (incidentally, I'm not sure how Sketchfab's tangent basis works, so I used the xNormal map).