Modeling a Gargoyle for Unreal EngineFriday, January 18 2019 by NATHAN MARSH
The intention of this article is to reveal my process of modeling a high detail asset for use in Unreal Engine. Game engines and realtime rendering are here now. Realtime applications promote the ability to view materials, lighting, animations, all in real time, measured in frames per second. With this ability, a different pipeline is needed than many animators and high resolution render artists are used to. PBR, or Photo-based Realistic Textures are paving the way to allow extremely accurate high detail textures to be baked into low polygon geometry. Some key components in the new pipeline are seen below, and this will be the outline for this tutorial1. Blocking/Layout Modeling
2. High Poly Sculpting
3. Low Poly Modeling and UV Mapping
4. Baking and Texturing in Substance Painter
5. Export to Unreal Engine ue4
1. Blocking/Layout Modeling
My biggest recommendation during the initial modeling phase is to use hotkeys for the program you are using. The layout phase should go fast and use of hotkeys will greatly speed up your performance. If you currently do not use hotkeys, at first, it may slow your modeling down, but this won't last longer than a session or two in the program. I began in 3ds Max but you can follow along in whatever modeling package you are comfortable using.
Use box modeling to create a basemesh that we can export to Zbrush for sculpting. The gargoyle will be created as three meshes; the gargoyle, a box top, and box bottom. Later on down the pipeline this equates to three texture sets and the ability to separate the parts for animation purposes for using the box and gargoyle independently.
Start with a Box in 3ds Max. Add an EditPoly Modifier on the stack. Press 2 to enter edge mode, then select a side edge and use hotkey alt + r to grab the ring of edges. Use Edit Edges → Connect and use 2 lines and pinch them out to match reference. (See image 1). Press 4 to enter face selection mode and select a face on the top edgeloop. Hold shift and select the next face in the edgeloop to select the entire edgeloop. Hold ctrl and select a face on the bottom edgeloop, then switch to hold Shift again and select the next part of this edgeloop. You should now have the top and bottom loops selected. (See image 2)
Go to Edit Polygons and select the options box for Extrude. Make sure the tool is set to local normals and drag out to match the reference. (See image 3).
Go to face selection mode again (hotkey:4) to select and delete the top polygon. Press 3 to enter Border Mode and click on the edge of the border. Press R and select the middle of the Scale tool to scale in slightly. Now press W to switch to the Move Tool. Hold down shift and drag down from your border edge to extrude it as deep as you want the box to be. Since we are still in Border mode, go to Edit Borders → Cap (See image 4).
Continue to use the process above and define the form of your model. I added another loop to the bottom of the model, extruded the edges, and built this out. If I need to delete an edgering, I select it in the same manner as before and then use ctrl + backspace to remove the ring along with it's vertices. At this point I have the general shape of the box. Now grab the middle face of the box and hold ctrl to select the middle face of each side of the box. Shift + E will enter Extrude and then click any of the selected faces to extrude all of them. Push them in by clicking right on one of the faces and dragging the mouse. (See image 5) I create a copy of the object with hotkey: ctrl + v to save before I start adding beveling and smoothing.
We will add supporting edges to our model to maintain the shape when we eventually apply Turbosmooth. Add another EditPoly Modifier on top of the previous one. You can rename it to "support edges" if you'd like. To put in support edges on a box, the easiest way is to use our orthographic views. Alt + W will toggle your 4 window view on. Drag a selection through all edge rings in the orthographic view, then use use Connect with 2 segments connected and pinch the edges toward the corners. (See images 6 and 7)
Now add a Turbosmooth Modifier. Set iterations to 2 and choose "Smoothing Groups" under Surface Parameters. This should give us a model with more geometry but maintain the the overall shape. If we get strange results, go back to the EditPoly Modifier beneath Turbosmooth, switch to face mode, select all faces, go to Polygon: Smoothing Groups. Hit "Clear All" and then "Auto Smooth set to 45.0. You can play with the angle to get proper smoothing groups in place. Put another Turbosmooth Modifier on top, set to 1 iteration and without smoothing groups enabled. This will smooth your object.
Create the top lid in the same manner as the bottom of the box. Start from a box, add an EditPoly Modifier to the stack, create extrusions and the basic shape. Add a second EditPoly Modifier to add support edgeloops. Add a Turbosmooth Modifier set to 1 or 2 iterations with Smooth By Smoothing Groups selected. Add a second Turbosmooth Modifier with 1 iteration. (See image 8).
We will now model a very quick basemesh for the gargoyle. I like to build out cylindrical objects using renderable splines. In the front viewport, select Create Splines → Line and draw out a line for the arm and a line for the leg. (See image 9)
Make sure to check "Enable in Viewport" so you can view as geometry, then add thickness and choose 8 sides. In the modifier stack, place an EditPoly to continue modeling in polygons. Select the faces at each end of the cylinders and delete them. Now we will create a box with 2 divisions for length, width, and height to use as the torso. Add some edges by selecting edge rings and connect. Press (hotkey: 1) to switch to vertex mode and press W. Select and move vertices to shape.(See image 10)
Grab the four faces near the shoulder and near the hip and delete them. We want to make sure there are 8 edges in each hole to match the 8 edges of our arm and leg. (See images 11 and 12)
For the arm socket and hip, grab the borders and extrude them slightly to give us an extra edgeloop in each area. Now we will make sure vertex snapping is ON by pressing hotkey: S, then right click the snap options box to set vertices as our snapping option. Snap vertices of the arm to the shoulder and the leg vertices to the hip. Once these are snapped into place, select all the vertices and go to Edit Vertices → Weld options and set your weld threshold to 0.01 to ensure we weld only the right vertices. (See image 13).
Make sure the mesh center axis is on the world center axis, and then add a Symmetry to the Modifier Stack, and then another EditPoly Modifier. (See images 14 and 15)
I extruded the top faces of the neck upward to create the head and extruded the feet out. At this point, cap any holes you have left in the mesh. (See image 16).
Export settings; For 3ds Max go to File → export as .obj and use Zbrush for the Preset.
This preset optimizes your mesh for use in Zbrush, as well as keeping axis alignment and world position correct.
2. High Poly Sculpting
While it is recommended to create a high poly object first, then duplicate and retopologize a low poly object, our blocking stage can also be effectively put into place to help us. Often times the blocking objects will become the low poly objects.
Once you become familiar with Zbrush and can navigate its interesting layout, it is an excellent tool for generating multiple meshes of differing Polygon counts from a single imported model.
Once Zbrush is open, Click "Load Tool" to load a Cube3d. Next, go to Import and find the .obj file you exported from Max. Drag the tool onto your canvas, then press "," to hide lightbox and "T" to edit your model. Go to the Subtool Menu and you will notice that your 3 meshes have been combined into one mesh. Let's split it into separate subtools now. Use Shift + F to view your polygroups. It should still be in 3 distinct polygroups, just like the 3 separate meshes in 3ds Max. Go to your subtools menu again and open the "Split" menu. Choose "Groups Split". We now have each subtool that we can sculpt independent of the others. (See image 17)
Select the Gargoyle subtool, go to the Geometry menu → Dynamesh → Dynamesh. The default settings will work fine at Resolution of 128. At this point, begin to sculpt the gargoyle with your Brushes. Re-Dynamesh by holding ctrl and dragging in your canvas ,whenever you start to see stretching happening. Build up details, and when you need more divisions, press ctrl + D to divide the mesh. Continue sculpting. Zbrush is a breeze once you learn just a handful of tools. My most used tools are below.
B → This brings up your Brushes
BCB → ClayBuildup Brush
BMV → Move Brush
BDS → DamienStandard Brush – Allows you to cut hard grooves into your surface
BH → hPolish Brush
BTD → TrimDynamic Brush → Trim away at edges, perfect for edgewear and rocks
* Holding down Shift with most brushes switches you to Smooth Brush
Continue to refine the form and sculpt. Use as much reference as possible. I was using images of gargoyles as well as anatomical references.
I added eyes, wings, nails, teeth, as separate subtools, and used the gizmo in ZBRUSH to translate the subtools into their correct positions. The rock feathers were created by making a new subtool , by inserting a cube3d in your subtool menu. I subdivided the mesh and used Trim Dynamic brush to sculpt the edges. Once I have the shape I need, I hold ctrl and drag a copy of the subtool. The new copy will still be active, so I scale, and translate it to where I need it. Once this is done, I go to Polygroups → Auto Groups. This will give separate Polygroup to each feather. (See image 18)
Next I remesh the gargoyle for cleaner topology. Go to Subtool → Duplicate to create a copy. Select the new subtool and go to Geometry → Zremesher. The default values worked well for me. You can select the target poly count you are trying to hit as well. These options you can play with to get different results. With the remeshed gargoyle active, go to Geometry → Divide. Now go up to your subtools and make sure your high detail dynameshed model is visible above your new remeshed subtool. Now Subtool → Project → Project All. This takes a moment to calculate and you will see the remeshed object start to adhere closer to the dynameshed model. Continue to subdivide your remeshed model, and reproject. I subdivided the remeshed model to 4 levels at 4.4 Million polygons. Whenever major changes are made to the model now, where we need to retain high detail on a new model, just remember this flow:
At this point, I am liking the sculpt and want to pose the gargoyle on top of the box. I used Transpose Master, which was finicky with my model. If I were to remake this model, I would probably build a quick rig for the gargoyle in 3ds Max or Maya to pose it, then re-import to ZBrush. Transpose Master left me with a bit of a mess on my mesh. (See image 19)
I fixed up and resculpted problem areas manually at this point, using smooth tool, and activating Sculptris mode. This helped me add detail on top of stretched geometry. This final sculpting phase for the gargoyle also allowed me to mask out fingers and toes to rotate and pose further. I also chose to run a Polish of the whole model by Deformation → Polish By Features. (See image 20)
Now we will go to our subtools for the box bottom. We will add a carving weave pattern to each side of the model and add some noise to the model. To create the celtic weave, Insert a new subtool → Plane3D. Hold shift and rotate so that we are in a front view. (See image 21)
Use hotkeys BCG → BrushCurveSnapSurface. Draw one half of the pattern, then use hotkeys BCM → Brush CurveMultiTube and click directly on your curve. A mesh will appear that follows your curve. If the size is incorrect, simply unDo and then you can adjust the focal shift and draw size. Once this is done, I duplicate the object and rotate into place. (See image 22)
I next refine edge loops on this new mesh to get a flat front face and harder edges. Use hotkeys BZM → Brush Zmodeler. Hover over an edgeloop. The tool will give you instructions on how to use Zmodeler. Hold alt and click the edge and the edgeloop will be removed. You can also bevel edgeloops by holding spacebar while hovering over an edge. In the menu that appears, choose Bevel and Edgeloop Complete. Let go of the Space Bar, click and drag the edge to bevel the edgeloop. (See image 23)
Once the weave pattern is in place, Merge the two subtools. Use BMT → Brush MoveTopological to push and pull where the weave strands should be intertwining. Next, move the whole pattern into place on the Box Bottom and then merge the box with the weave design. Now we will dynamesh this object and do some smoothing, remeshing, subdividing, and projecting. We will do this for each side of the box where a pattern should go. (See image 24)
Add some noise and sculpting to the Box top as well.
Final High Poly Sculpt. (See image 25)
3. Low Polygon Modeling and UV Mapping
The low poly mesh creation is fairly simple since we are working on a static mesh rather than a character that needs to animate and deform. The technique we used earlier will come in handy again. Remesh → Subdivide → Project. We will subdivide the low poly model high as well so that the lowest subdivision will conform properly to the high poly. Another method that we can use in conjunction with this is Decimation Master.
Once this has been pre-processed, select Decimate Current. The default setting worked well for me, but feel free to adjust sliders and get the right level of decimation that holds the shape and detail of your model, with as few polygons as possible.
I've remeshed and decimated each piece of my model. Now is a good time to rename our objects so we can be consistent and not drown in objects named highpolyGargoyle05_highFinal02 etc. When we use Substance Painter to bake details into our mesh later on, naming convention is very important. Each object should have a name that matches with it's counterpart, except the suffix _high or _low. Here are how my objects were named:
One last step before baking our high poly to our low poly: UV Mapping the lowpoly model. I exported each one of these objects from Zbrush as .obj and then imported all of my lowpoly objects into Maya in order to UV. I prefer Maya for UV over 3ds Max, but there are a host of programs you can do UVs in. UVs can be done cheap and easy or meticulously trying to capture every pixel of texture density as possible. I switched to Maya for the UV layout since the automatic flatten map seems to work well. I used this for the Box and the BoxTop. For the Gargoyle I planned seams, cut them, and relaxed the Uvs to give me nice UV islands.
No overlapping UV edges or points
Try to have straightened UV shells as much as possible.
Any edge that has a 90° angle between faces should have it's UV edge cut for your UV map
In 3ds Max or Maya make sure each sub object of your model has Uvs applied, and assign a unique material to each. Name the material what you would like Substance Painter to show in its' texture set list. If it isn't already done, select all your edges that should be smooth and smooth normals. In 3DS Max, assign smoothing groups to define hard and smooth edges. Select all parts of the whole model and export as a .fbx file. Under the "Include" options, I choose Smoothing Groups and uncheck every other item under Geometry. All other options can be unchecked or left alone.Final low poly asset. (See image 26)
4. Baking and Texturing in Substance Painter
While I have not used Substance Painter for very long, the logic and organization behind it makes it feel very familiar and approachable.We will cover some basics of Substance Painter. Here are some tips to keep in mind. If we have multiple parts of a model, you can choose to use multiple texture sets or combine the assets into one mesh with sub objects. The sub objects can be assigned their own materials. Every piece of low poly geometry will need Uvs and a unique material assigned
Use PBR – Metallic Roughness (allegorithmic).
Next to Mesh, click "Select" and navigate to the low poly mesh we saved as .fbx.
Leave "Create a texture set per UDIM tile" unchecked since we are using 0 to 1 UV space only.
Normal map format I use DirectX
Check "Compute tangent space per fragment" since Unreal Engine requires this to be set.
Document Resolution: 1024 This can be changed laterHit "OK" (See image 27)
Now let's go to our Texture Set Settings and click Bake Mesh Maps. Leave output size at 1024 for now. Click the button next to the field for High Definition Meshes and load our high poly .obj files. Scroll down to "Match" and change it to "By Mesh Name". Click on "Bake all texture sets". You will know relatively quickly if the bake works or not. If it fails, we need to check our naming conventions.
The bake was successful, so we can dial up our settings and rebake the mesh. Change output size to 4096 and Antialiasing to Subsampling 4x4. Click Bake all texture sets. (See image 28)
I experimented with combining Smart Materials and using layers to build up the moss on top of the rock texture. I worked on the Gargoyle texture set at first and then instantiated the material across texture sets. The great thing about Substance Painter, and its main draw in my opinion, is that what you see is truly what you get. Experiment with various materials, smart materials, and masks. Play with generators that can find edges of your model. You can get differing effects simply by changing your Texture projection type from UV projection to Tri-planar projection. This can also cover up slightly for quick n' dirty UV mapping. (See image 29)
File → Export Mesh → File name: I name mine "SM_BoxGargoyle.obj"
File → Export Texture Files, Set all texture sets to 4096x4096
Unreal uses the Metallic PBR model, so make sure this is set before exporting our texture files. (see image 30)
5. Import to Unreal Engine ue4
If you don't have an Epic account, go and get one. It's FREE.
Open up the Unreal Project Browser and click on New Project. Use a Blank Blueprint and make sure "No Starter Content" is selected. Name the project and select where you want to start your project. Click "Create Project". In your Content Browser, create three folders by right clicking and creating Folder, or choosing "Add New".
Click on the Content Folder and then click Import. Select the low poly mesh. Once this is imported go to the folder that we saved our texture files to. Select all of them and simply drag into the Content Browser in Unreal. Move the material files to Materials Folder, textures in your Texture Folder, and Your mesh goes in Props folder.
The last step is to connect all of the texture files as nodes to the correct materials. Double-click the Material to open up the Material node connections. Drag and drop your texture files with the same name into this browser. If you click a texture node, its full name will be revealed in the Details Window.
Make sure to click "Apply" at the top of the Material Window and these changes will take affect. Repeat this for all three texture sets. Select all three meshes in the Content Browser and drag into your scene. You may be prompted to save at this point. (See image 32)
Hit Play to see how everything looks and runs.
If you've made it this far, Congratulations. I hope you can pull some information out of this tutorial that can help you and lead further to success.