Oh man.... never give a programmer the chance to talk about the stuff he's working on.....

Making something data-driven after it's done is much harder, hehe. A lot of what I'm going to talk about is related to our AI system since that's my main interest and the system I know the best. First, I'll explain how the AI system works so you have a high-level understanding of it. Then I'll show you the way we make it all datadriven and generic, mostly using XML. If I have time I'll go into our Property system, which is simply awesome. You can define whole hierarchies of properties in XML.

We call our AI system the UtilEcon (short for Utility Economy) system. It works by having agents (Util Units) that have needs and a desire to fulfill those needs. They do so by seeking out Econ Units and arranging a transaction. Furthermore, Util Units can also satisfy the needs of other Util Units just like an Econ Unit (in reality, UtilUnit is a class that inherits from EconUnit).

So how do we get all this flowing? First, we define satisfiers. A satisfier is something a UtilUnit needs, and that EconUnits can provide. They are the raw needs that exist. In Barbie, for example, we have satisfiers like "socialize" and "gossip" and "learn". Ratrace doesn't have learn, but it does has work and romance. The point is, the satisfiers are defined in XML and change for each game. Only the concept of a satisfier is in C++.

Great! Next on the chopping block are goods. Goods are the tangible form of satisfiers. For example, in Rat Race we have both hunger and thirst. Some things, like fruit, provide both. Coffee satisfies both thirst and caffiene. Goods are also defined in XML and can relate to multiple satisfiers. Here's an example of one from Rat Race:



Again, even the names are defined in XML and only the concept of a good is known to the code.

Next up we have transactions. This is how the goods (and ultimately the satisfiers) get from the EconUnit to the UtilUnit. All they really do is define the specific transaction that will take place, and the good involved. Here's one from Barbie:



The context is used by our animation system to figure out which animation to run. It will look at the actor and the receiver and choose the appropriate "Use" animation in this case (all that's datafied too, but I'll save it for a different lecture). So, each transaction is mapped to an animation context and a good; the goods that were defined above.

That's all we need to start defining EconUnits. Let's look at a very simple one:



Does this look familair? It's the BookShelves EconUnit that used the LearnAtBookShelves transaction. As you can see, everything is once again defined in the data. The name, capacity, and the goods it provides.

Util Units are a bit trickier. We allow the definition of templates to make things easier. Each UtilUnit has a list of plans, which are really nothing more than maps to transactions that util unit will attempt to perform. When you define the actual UtilUnit, you give it a template and add to or override what's there.

Every UtilUnit has Goods that it can provide, just like an EconUnit. Traditionally, these are all social goods, like "humor" and "socialize". The transaction type is <exchangememe> instead of <gather>, which (basically) tells them to start talking and what to talk about. They scan the speech database and dynamically generate an appropriate topic and response (VERY cool, btw). Anyway, they also define their drives (a mapping of satisfiers they want along with how much they want them). And finally, they define their plans. Plans just map to transaction types. Here's a majorly stripped down UtilUnit:



And that's how the AI system works (basically). Now, how does this look in C++? Most of the items that I've discussed map directly to classes. We have a Plan class and a GoodAmtMap class and of course EconUnit and UtilUnit classes. All of this data drives the construction of our AI objects. It finds a UtilUnit named Courtney so it news up a UtilUnit object. Then it loads all the plans found for Courtney to build her list, etc. The code only really knows the concepts of things. It knows what a good is, and it expects to see a list of satisifers with floats attached. It knows how to use those. The key here is that this frees the level designer to screw around with this stuff without having to worry about recompiling. In fact, if I want to make a new game with totally different goods and satisfiers, I can do that without touching one line of C++.

Now, how about Lua? The beautiful thing about Lua is how easily it integrates into the game engine. Take a look at tolua++. That's what we use. If I want to expose a function, I do this:



That's it. Bam, it's in Lua ready to be called. We to_lua the hell out of everything. to_lua++ lets you expose objects to lua as well. For example, we have a global singleton called UtilEconMgr that manages the UtilUnit and EconUnit maps, among many other things. Since it's global and to_lua'ed, I can access it in Lua by name. I can also call any to_lua'ed function:



That assigns a pointer to the variable uuCourtney. Now I can call UtilUnit member functions through that pointer:



We get crazy with this, in our property system. All entities and properties are actually created in Lua by calling new.

(( EDIT: Making new post ))

Okay, on to the Property system. Basically, anything that can doing something useful is an Entity. Now, Entities can't do anything useful by themselves, they are only blank slates, or containers. The useful stuff are the Properties that attach to these entities. Every character and object in the world (anything that's not static geometry) is an Entity. We have Hard Properties and Soft Properties; hard properties are defined in C++ while soft properties are in XML. A hard property is something that requires the engine, like ShadowCaster or Relator. Core systems are often accessed through properties too, like Converser or Animator.

Now, soft properties are all XML and/or Lua. They can inherit hard properties or use them and often do, but they don't have to. If all you want to do is encapsulate some data, you don't need to inherit from anything. Soft properties contain STL maps that map variable names to the values they contain, so in XML you create the variables that these properties have.

In my example, I'll use the Relator property since I wrote it and know it well. It also relates to the AI stuff we were just talking about. The Relator property is a hard property implemented in C++. It's purpose is to give the Entity the ability to form relationships. At this point, it acts more like a manager than anything else, but that will change once I get the "social flocking" working (that's gonna be awesome ). Anyway, it ties into the speech system and UtilEcon system to determine who the Entity wants to talk to next and how successful they will be. It currently does this by aggregating the appropriate sub-entities attached to it's parent (remember, entities are just containers for properties), doing some math, then returning a modifier.

The first such sub-entity is the Aptitudes entity, which we attach to any other entity. (An aptitude is basically a stat, like Charm or Guts.) This sub-entity has any number of soft properties, each with a single integer value. They can be named anything you want and you can assign any value to them, which the Relator component uses when we determine the success or failure of some dialog. If Marty (a dumb guy) tries to talk to Ron (a genius) about battle robots, Marty is almost certainly going to fail because his low Smarts stat. Here's the XML:



Internally, this is just an STL map, using strings as the key (the variable name). We actually do some magic to ensure the right type is used, but under the covers it's just a map. All the speech tables, which are also defined in data, have the ability to call prereq's for each possible line that the NPC might say. One of the prereqs can reference an aptitude by name and, if found, it will be used in the calculation by Relator when determining success or failure.

The next entity is the FriendshipMatrix. This is used by the UtilEcon system when determining who they should talk to. The idea is that you will talk to your friends more often than people you do not like. Thus, when scanning the world for possible receivers (EconUnit's), the Relator queries this table to see if we have an opinion about this person. I chose to implement this table in Lua instead of XML. In our system, XML is used exclusively for initialization while Lua variables are used for data that changes over time. Friendship is fluid, so I chose to make the table in Lua so that friendships can form and die over the course of the game. Right now, Barbie is the only game that uses the FriendshipMatrix, and it never changes. On a side note, I'm the only one who really understands the table at this point, which just goes to show you that a good tool is key to getting the level designer to do a good job. As it stands, it would fall on me.

The table is simple enough, it's really just a multiplier that Relator gives to UtilEcon when determining a good target to talk to. The table is bi-directional, meaning I could like you a lot but you could hate me. In that case, you would often be the receiver of my speech, but you would rarely come and talk to me. The idea there is that you can get a lot of perceived behavior depending on how the dialog is written.

The last thing I want to talk about before falling over into unconciousness is the EXE. We're making two games right now. How many EXE's do we have? Just one. That's right, if I want to run barbie, I run the EXE with "-title Barbie" on the command line (to tell it which data directory to use). For Rat Race, it's "-title Ratrace". No recompilation needed. That's the goal you should shoot for. Can you make two games with the same engine without recompiling? If so, you've achieved your goal.

That having been said, Rat Race is horribly broken right now. The reality is that we're changing code every day, and that code efffects the data we're reading in. Maybe we changed Loyalty to be an XML attribute instead of it's own element. Minor tweaks can have unforseen consequences. Since we never play Rat Race anymore, months worth of changes go by and all of sudden the lead engineer is spending two weeks trying to get past the first frame.

Well... I do believe that is longest most rambly speech I've ever given. I hope it helps in some small way with whatever you're working on. If you have any specific questions, need more detail or clarification, or just want to call me a crazy purple haired freak, feel free.

-Rez
PS: My apologies for grammatical or spelling errors.

That's some good stuff, Rez. Here's a question for ya:

Why use XML instead of Lua as your data definition format?

Since you're already using Lua for scripting, you could also use Lua to describe nested properties in exactly the same way you are using XML, no?

Is it because XML offers the functionality of schemas? Is it because there are some excellent XML-editing tools out there?

I do feel that a hard separation between data definition and scripting is necessary, and using Lua for both does make it easier to shoot yourself in the foot (same tech applied for different purposes). Using XML for data definitions (as you have done) definitely saves you the from having to constantly explain why Lua data files should be kept in a separate directory from the Lua script files.

But still, what does XML offer that Lua does not?

That's a good question, Kain. There are several benefits of XML, several of which you already named. We use Schemas for each of our major sub-systems and we have several tools that generate XML for us to help minimize the data crunching.

I haven't actually asked anyone, but I think the real reason we still use XML is because the company has used it for about 3 - 4 years now. It's so integrated into our engine that ripping it all out would be huge task and there's really no reason for us to do it. There wouldn't be any functionality gained in most cases.

Interestingly enough, our save/load file format is an XML/Lua hybrid. When saving, we dynamically generate a bunch of Lua calls that will get the game world to the place it should be and set everyone's position and orientation. We also load a bunch of XML that creates and sets up other objects (like UtilEcon and our speech system). Interestingly enough, it's the older systems that use mostly XML and the newr systems that are mostly Lua.

-Rez

Thanks for the explanation, Rez. I've been told to keep data separate from scripting to avoid headaches down the line, and that is exactly what you are using.

Tarviathun, I'm a bit unclear on your question, but I'll try to answer as best I can... Applying the logic of property data associated with dynamic objects in a modular design for the engine code is one of those things that haven't quite been standardized just yet. Assuming your game has the same parameters as Teapot wars, you won't be able to avoid data duplication (such as position) with authoritative data existing in one system (physics). This whole thing basically comes down to load and save functionality. When you design the load/save component of your system, your main goal is to have it be easily manageable by humans by minimizing the chance of forgetting to serialize a certain detail.

My suggestion: Use WorldStateManager. Essentially, this class would be responsible for loading a world state and saving a world state. The dynamic property-laden olbjects that make up the world state will be read by this class as inert data (let's call this cObjectData) so that WorldStateManager can send out a CreateObject event that contains a cObjectData member that one or more game components (physics, AI, sound) can read and initialize their own relevant object to. Each component only cares about a subset of the data available in the cObjectData. Also, this cObjectData is inert data, meaning that it is only used during load. Throughout the course of gameplay, the data does not need to be kept around as it will only waste RAM and become grossly out of synch. During save, all components will have the neccesary query methods that will allow WoldStatemanager to reconstruct the property information for alll dynamic objects in the world so that it may be serialized into a file.

Note that on a console, you want the format of this save/load data to be as memory-mapped as possible because storage space can be limited (They really want you to be able to fit things on a memory card) and people judge games by load times. This means that some of the suggestions I've described above would have to be broken to reduce load times (like the suggestion of using the event system).