Streams and Messages

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • Streams and Messages

      I am currious about a couple of things both conserning streams. They are related to the book alittle I guess.

      1. The book mentions using streams to init objects, why is this a good thing? is it just so the whole object setup approach is reduced?

      2. Serializing messages. They way I've done messages in the past was awful void* and enums, the stream method seems to be very elegant, but surly when there are 100 or so messages each tick this is an unnecessary overhead?
    • RE: Streams and Messages

      1) A stream is basically a smart "file" you access that can act as a bucket of bytes to read from and write to. I have file in quotes there because the source doesn't have to be a file. The concept of a stream is totally abstract, so in one instance it can be a configuration file, in another it can be from loading a save file, and in another it can be a data chunk you get from the network or some other place. You have a single, abstract interface that doesn't care where the bytes come from. You can change the source of the data without changing the way you read it. The same thing goes for writing. There are other advantages, but that's probably the biggest one.

      2) What kind of messages? And what do you mean by serializing? To me, serialization typically means writing something to the hard drive; ie serializing an object means writing that object to a file. That's probably just from being in Sims land for so long (all of our save data classes have some form "serialize" in the name). Are you talking about serializing out to the network? If so, that sounds like a great place for a stream. If you're eating a lot for overhead, maybe you can batch them and send out large chunks, maybe even one single batch per frame. Then again, I'm not a network engineer so someone more knowledgeable in that specific area may tell me that I'm full of crap. ;) It seems reasonable though.

      -Rez
    • Thanks for the description of stream.

      When I said message I think I should have said event data. The book mentions to look at using boost::serialize to send messages/events, I thought this might have abit to much overhead (I've not tested that tho). My last game (student game we are talking about) used a very fast/loose method.

      eventHandler(unsigned int msg, void* data)
      {
      if(msg == PLAYER_MOVED) {
      Data* data;
      static_cast<SomeData*>(data);
      }
      }

      perhaps I misunderstood the book but I think it was suggesting sending the stream.

      eventHandler(unsigned int msg, std::stringstream stream)
      {
      if(msg == PLAYER_MOVED) {
      Data data;
      data.initWithStream(stream);
      }
      }
    • Ah, events! Okay, now I get what you're saying.

      Streams can work, but it really depends on how you use messages. On The Sims Medieval, we had multiple layers of inheritance based on common things we sent messages about and registered a delegate function to handle them. There were actor events, for example, that would include a reference to the actor that the event was about. This allowed a relatively common interface for common types of events. If the majority of your events have similar data, that kind of method may work.

      For Game Coding Complete, the idea is that you have an event manager that you use to send events and register listeners. Each event type had a unique id that listeners would use when registering to listen for specific events. There's an Event base class that you typically inherit real events from. These sub classes contain all the data you need and are sent to the listeners when you fire off the event. The listeners implement a ReceiveEvent() function and get a pointer to the base Event class, which they cast to the appropriate sub class. This is perfectly fine because the listeners already have to know about the specific event class to register as a listener for that event. The sender of the event also has to know about it so they can instantiate the concrete sub class. No other system, including the event manager, needs to have any idea about the event sub class.

      Does that make sense? It's 3am over here and I've been coding since 10am. I'm getting a little loopy. ;)

      -Rez
    • Thanks for that.

      The last application everything riled on events (other than drawing).

      I am currently modifying my event manager (which is similar to Game Coding Complete) to be very simple for prototypes, which is what I do a lot of at university. All the static_casting looked abit messy and seems to polarise various lectures opinions.

      If streams are good enough for this I will stick with them for this as the seem to clean my code up slightly.

      Happy Crunching.
    • Sorry to clarify streams are ok for events but better just sending the pointers.

      I have a problem with queued up events. I am reluctant to create lots of messages on the heap. Instant messages are fine, but queuing up messages are somewhat of a problem, in a test on the PC, I've been using serializing them to a string stream and storing them as a std::string, and then re-serializing them when blitting the queue, this seems overly verbose.

      I should also mention that my target platform is PS2 Linux, with a 10 year old compiler (My university are reluctant to updated the compiler incase it breaks stuff, and Sony have giving us only two PS3's dev tools) Lots of allocations during the update cause easily noticeable frame-rate drops.
    • So, just to get a little background here, you're making a game and networking two PS2's together that are running Linux? What kind of game are we talking about here? How many messages are you sending per frame? How big are the packets? What data are you sending? What's your worst-case network speed? Have you built and/or profiled any real use-cases yet?

      The key is to figure out where the real bottleneck of the program is. I'd build a vertical slice of the game that has the same number of entities, area size, and network traffic that you'd expect and then profile to see where you're really spending your time.

      While working on The Sims Medieval, we were surprised that a large part of the AI budget was going to one function: GetNumberOfBuildings(). Turns out that this function was doing a really inefficient O(n) operation every time on a giant data set and I think it even hit the hard drive to read in the data. We weren't caching it anywhere. We would just call the function in MANY places. The fix was trivial and it never showed up again.

      My point is, you can guess all you want, but you'll never know what's really a problem until you actually profile it.

      Oh, one last thing: you SHOULDN'T be creating messages on the heap. You should build a memory pool and use that for creating messages, ESPECIALLY on a console.

      -Rez
    • I was speaking more generally. Using streams for events internally as well as over the network.

      However networked events will occur as well. It is simple multiplayer ww2 flight sim game where each client will send the posiiton and settings of that players planes. And it will also receive the positions of the other players.

      The last test I did was with two PS2's networked together but I think I'll move this to a dedicated machine to allow 6 or 8 players.

      I am trying to make a generalised solution, as time is generally my enemy.

      Just to be sure...
      Instant message data can sit on the stack.
      Queued message data can go into the pool.
      Networked message data gets serialized and sent. (revived and de-serialized)

      I have been avoiding pools but I guess the time as come. Looking into pools they seem like the right solution.

      Side note on pools.
      Would they be useful for game objects, or are they more suited for smaller data? In the past I've been creating all the game objects and storing them. When an instance of the game object is needed one is taken from the list and activated so to speak. and then returned when it is no longer needed, thus not requiring to create and destroy on the fly.
    • Creating a client/server architecture makes the problem really difficult. What lives on the server? What lives on the client? What are you sending over? Keep in mind that you can be sending data every frame but it's not going to be updated that fast. Go to a cmd window and type ping google.com. It'll send a few 32 byte packets to google and wait for a reply, telling you how long it took. Mine took around 18ms each time, but one time took about 44ms. That's longer than a typical frame 30 fps (which would be 33.3ms). What do you do in those cases? There are a number of possible solutions but when you're dealing with a network, you have to be prepared for lag, dropped packets, dropped bytes (ie corrupt data), dropped connections, etc. You have to figure out how to handle all this stuff gracefully. On top of all that, you have to actually write the game. This is why companies tend to have a team dedicated to it (we have an entire server team which is at least as big as the gameplay team and much bigger then the client team).

      How long do you have for this project? What are you envisioning the end product actually looking like? What will the player be able to do? What's done already?

      Pools are AWESOME. I use them for just about every volatile small allocation I do. I generally use them in two ways. First, I create a number of generic pools of various sizes from about 8 bytes on up to around 128 or 256 bytes and use them for all my random allocations. The pools can grow if need be and once I have my game mostly done, I can tune the individual bucket sizes so that the pools never have to grow, they're always of the right size.

      The second kind of pool I create are pools of specific objects with a static size. For example, I create a pool specifically for pathing nodes. When I was at Slipgate, I used a memory pool to hold the render data for billboards used for player and enemy life totals.

      They start to break down for larger allocations because the memory you waste ends up overriding the performance benefits. A quick look at my own home-grown engine reveals that I cap it out at 128 bytes. Anything larger goes to a different memory manager that's optimized for larger chunks.

      Regarding memory pools, make sure you have some kind of logging. You'll absolutely have to tune them. For example, here's the output of mine on shutdown:

      Source Code

      1. Destroying memory pool: [MouseInputMessage:60] = 1/4 (180 bytes wasted)
      2. Destroying memory pool: [KeyboardInputMessage:44] = 1/4 (132 bytes wasted)
      3. Destroying memory pool: [SpriteRenderCommand:48] = 638/1024 (18528 bytes wasted)
      4. Destroying memory pool: [ColoredSpriteRenderCommand:52] = 0/128 (6656 bytes wasted)
      5. Destroying memory pool: [RefCount:8] = 11/1024 (8104 bytes wasted)
      6. Destroying memory pool: [Main Thread Allocator:8] = 0/32 (256 bytes wasted)
      7. Destroying memory pool: [Main Thread Allocator:16] = 22/32 (160 bytes wasted)
      8. Destroying memory pool: [Main Thread Allocator:24] = 2/32 (720 bytes wasted)
      9. Destroying memory pool: [Main Thread Allocator:32] = 14/32 (576 bytes wasted)
      10. Destroying memory pool: [Main Thread Allocator:40] = 0/32 (1280 bytes wasted)
      11. Destroying memory pool: [Main Thread Allocator:48] = 7/32 (1200 bytes wasted)
      12. Destroying memory pool: [Main Thread Allocator:56] = 2/32 (1680 bytes wasted)
      13. Destroying memory pool: [Main Thread Allocator:64] = 0/32 (2048 bytes wasted)
      14. Destroying memory pool: [Main Thread Allocator:72] = 0/32 (2304 bytes wasted)
      15. Destroying memory pool: [Main Thread Allocator:80] = 0/32 (2560 bytes wasted)
      16. Destroying memory pool: [Main Thread Allocator:88] = 0/32 (2816 bytes wasted)
      17. Destroying memory pool: [Main Thread Allocator:96] = 0/32 (3072 bytes wasted)
      18. Destroying memory pool: [Main Thread Allocator:104] = 0/32 (3328 bytes wasted)
      19. Destroying memory pool: [Main Thread Allocator:112] = 0/32 (3584 bytes wasted)
      20. Destroying memory pool: [Main Thread Allocator:120] = 0/32 (3840 bytes wasted)
      21. Destroying memory pool: [Main Thread Allocator:128] = 0/32 (4096 bytes wasted)
      Display All


      Glancing at this, I can easily see how many bytes I'm wasting and which pools need to be tweaked (or removed). The Main Thread Allocator pools are the generic ones I spoke of. They're typically used in allocators for STL data structures. Everything else is tailored to a specific purpose.

      -Rez
    • Bascially I was going to have each client send its position/status to the server each time there was a change in input. And have a dead dead reckoning algotirthm try and predict the locations until a new message has been received. I don't have time to make this amazing I know.

      We get accessed on the tech in our games rather than the game overall, I want to focus on networking and data-driven objects, extend my event system to be able to queue events and of course transmit them now.

      I have most of the framework done, events, models, audio, and a basic network layer that is still needs to be completed. its always incomplete at uni, but I think the focus is prototyping rather than complete games at this stage. I have until December (not exclusively tho).

      Thanks for all your help, your little insights into industry are really helpful.