139. Importing full 3D models into Codea

This post (or posts) explains how we can import full 3D models into Codea. If this interests you, or if you just like journeys of exploration and making design decisions, read on.

Until now, I thought it was virtually impossible to import 3D models into Codea – by that, I mean models that you can view from any angle, like the aeroplane above. You can build these models in applications like Maya and Blender, and they are made up of thousands of vertices and maybe many texture images.

So I’ve either been drawing very simple 3D objects like rectangular buildings, or “billboarding” – which means using 2D images (like trees) and rotating them to always face the viewer, so you don’t ever see that they are just flat images. Obviously, this has serious limitations!

Model datafiles

To get a real 3D model into Codea, we need (someone else) to create (and make freely available) a list of vertex points (vec3), texture positions (vec2), and texture images. Maybe we can also get normal values (for lighting).

There are plenty of free models donated by generous people. Unfortunately, the datafiles produced by most modelling software are in binary form, and impossible to read without a modelling program.

However, there is one exception. The OBJ file format is in plain text and is machine readable (meaning Codea can read it). And using Blender (free!), we can convert models from most other formats to OBJ. So, subject to any licensing restrictions on the models, we are in business.

The design issues

A number of design issues emerged during this project

• how to read the data from OBJ files
• how and where to store this data (and texture images) so users can download them
• where (and in what form) to store the downloaded data

Now suppose for the moment that we know how to read OBJ (I’ll come back to this, later). Let’s look at the other issues.

We will get lists of vertices and texture mappings from the datafile. What should we do with them, and with the images that are used for texture mapping?

Data storage

Storing pre-processed data in Codea

We could use a program outside Codea to “pre-process” the OBJ data, to get it in the right format for Codea, and paste the data into a Codea tab, ie “embed” the model data.

This is what I started with.

However,

• it is difficult to find a program that everyone can use,
• pre-processing adds an extra step, and
• storing data in Codea can overload it, especially when models get large.

It’s not a great solution.

So I came round to using Codea to directly read OBJ data and convert it to what it needs.

Storing raw data and images on the internet

Now I have to decide where the OBJ data (and images) are going to come from. I should point out that when you find models on the net, they are usually zipped, so I can’t just link to a zipfile! That makes it unavoidable that if you want to share a project using 3D models, that you have to store the OBJ data and images somewhere they can be downloaded.

I obviously considered things like Dropbox, but the problem is that it doesn’t seem to give you direct links, ie if you share a textfile, and someone clicks the link, they get a webpage with a download button, not the text. Similarly, if you store images in Flickr, clicking a link gives you a webpage and not just the image.

Finally, I settled on using gists (gist.github.com) to store the OBJ text data, and Photobucket to store the images, because both of these offer direct links. (There are some tricks, though. Having created a gist, don’t take that resulting URL. Instead,look just above and to the right of your text for the icon with two angle brackets (or a button labelled “Raw”). Click that to get the link to the pure text).

So now we have all this data waiting to be downloaded and converted into something amazing.

Downloading data and images asynchronously

Our next problem is that we may need to download 10-20 image files for a model. Codea can retrieve data from the internet, but it does it asynchronously – which means you ask for a file, and give Codea a function to run when the file is retrieved. In the meantime your code keeps running! So how do we get Codea to ask for the files one at a time, waiting for each one to download before asking for the next one, and not drawing the model until everything is downloaded?

Fortunately, I already answered that here. I used a version of that approach.

Storing data and images

Originally, I wanted to store the data and images in a Dropbox subfolder, which would allow users to edit the folder directly from PC or Mac. However, there is the small problem that this doesn’t seem to be possible.

There is another problem, that the OBJ data file doesn’t have an image file extension like JPG, so it doesn’t appear in sprite lists within Codea. This means it would be impossible to delete old data unless you knew the exact names that had been used for the datafiles.

What I settled on was using saveGlobalData to store the text OBJ data (because it is fast, and there is a function that gives you a list of all saved items, making editing fairly easy). And the data is out of sight in the meantime.

I decided to store the images in Documents, but to prefix them with “z3D” so they get sorted to the very bottom, and don’t make a mess.

So when you first load a model, Codea will download and store the OBJ data and images. This means next time, they will be loaded directly from your iPad.

I think this is a good solution. It keeps the code small, and minimises the effect of storing all the data for a model.

I haven’t answered the question of whether it’s better to store the original OBJ data, or pull out what you need and just store that. For the moment, I’m storing the original OBJ data, but that might change.

Reading the data file(s)

This is the hardest part.

OBJ files come in two separate files

• a .mtl file that lists all the “materials” used
• an .obj file with the vertices, texture mappings etc

The mtl file

A “material” is just a group of settings for light colour, texture mapping etc, that all apply to part of the model. So if you have a plane with two wheels, you can define a material called “wheel”, and then use it for each group of vertices making up a wheel. The only purpose of the  mtl file is to define all the materials used in the obj file.

Here is a sample material definition

newmtl F4U1GLS
Ns 96.078431
Ka 0.000000 0.000000 0.000000
Kd 0.000000 0.269804 0.153725
Ks 0.500000 0.500000 0.500000
Ni 1.000000
d 0.200000
map_Kd /Maps/Underblue.jpg
map_bump F-117_Nighthawk_N.png
illum 2

newmtl is a keyword saying that this is a new material, and is followed by the name (that will be used to assign this material to vertices later on).

Ns is the specular exponent, while Ka, Kd and Ks are the rgb settings for ambient, diffuse and specular light.

map sets the texture image and contains the file path and name. There can be separate images for ambient, diffuse and specular light, and bump maps too, each of them being labelled by adding Ka, Kd, Ks or bump to the word map.

We need to change any lines marked map because

• at the moment, the Codea code only handles one texture per material, and not bump maps
• we are going to save it in Documents and may want our own name for it (eg add a prefix with the model name, so it doesn’t write over other image files for other models), and
• we need an internet URL where it can be downloaded, not a path on the local computer

So the adjusted line looks like map filename url, eg

map texture9.jpg http://i1303.photobucket.com/.../texture9_zps0f22ad7c.jpg

Note – it doesn’t matter if you have map, map_Ka or map_KD at the front of the line – my program just looks for the line to start with “map”, and uses spaces as delimiters to find the file name and path.

I don’t worry about the other settings at this point.

The obj file

The obj file looks something like this (omitting nearly all the data!)

v 2.578325 1.572789 -1.550868
v 2.578325 1.572789 -1.550868
v 2.578325 1.533261 -1.522983
v 2.578325 1.479815 -1.470769

vt 0.145067 0.156179
vt 0.145067 0.156179
vt 0.145067 0.202403
vt 0.145067 0.259817

usemtl bombs_2
f 10/10 17/17 19/19
f 19/19 11/11 10/10
f 11/11 19/19 20/20
f 20/20 12/12 11/11

Lines starting with v are points in the wireframe of the model, and each contains an x, y and z value.

Lines starting with vt are texture mappings, and each contains x and y co-ordinates for the corresponding vertex (ie the fifth vt applies to the fifth v).

Lines starting with f are “faces”, or polygons, groups of points. You’ll see above that each f line has three pairs of numbers separated by a slash. The first number in each pair applies to wireframe points, and the second applies to vertex mappings, and each of these (three) pairs is a vertex in a mesh. There are three vertices per line above, so each line defines one triangle in the mesh.

The numbers refer back to the v and vt lists above, so looking at the first f line above, the first vertex has 10/10. This means that it uses the 10th wireframe point in the v list, and the 10th texture mapping in the vt list. Often the two numbers are the same, but not always.

Sometimes, the obj file also contains normal values for each vertex, in which case those values are listed with a prefix of vn, and the f lines have a third number separated by a slash, eg 10/10/5.

Note also the line usemtl bombs_2 just before the list of f values. This tells us to use the material called bombs_2 , as defined in the mtl file. This is how we know which image to attach to which vertices. Obviously, we need to create separate meshes for each of these groups of “faces”, if they have different images.

There is one little complication. List of faces can have more than 3 vertices – in fact, the largest number I’ve seen on one line so far was over 60! Codea can only handle triangles, ie 3 faces, so what do we do?

Initially I simply calculated the “centre” of the face, being the average of all the vertices, and then created triangles consisting of each pair of vertics together with this centre vertex. So a face with 60 vertices will end up with 60 triangles. However, this can lead to flickering and unpredictable results if any of the resulting triangles overlap. Better is to go back to Blender and set it to convert all faces (groups of vertices) to triangles before exporting as an obj file.

Combining the mtl and obj file

There’s no real need to have two separate files, so I add the mtl file to the top of the obj file, and separate them with the tag [obj].

The result

A good model can produce a stunning result, like the aeroplane at the top of this post, or the Spitfire below.

Here is some code containing a number of models you can view. If you look in the Assets tab, and open up one of the URLs there, you’ll see what one of the mtl+obj files looks like.

But what makes all this work really worthwhile is being able to use one of these models in a game. Below, I’ve included a model of the Apollo lunar lander in my 3D moon landing game. Previously, I had a yellow marker in place of the lander, after landing on the surface, but now I can put a great model of the lander there – and put it in the picture while flying as well, as shown below.

You may wonder why I keep chosing aeroplanes for models.

• They don’t need animation – if I used living things, their arms and legs and heads would have to move, and I’m not ready for that!
• You don’t need to draw detailed surroundings for aeroplanes, just a hint of clouds and ground below

My next step is to try to fly these planes in Codea. To do that, I have to learn how to use something called a quaternion, that I don’t understand at all. It seems to me that the more you learn, the more there is to learn…..