resource cache questions

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • resource cache questions

      In GCC's xml resource loader, the raw buffer containing the xml file is discarded. However, an empty buffer for the processed data is allocated with the same capacity as that for the raw xml.

      After the xml has been parsed, the handle still retains this empty buffer to processed data. Shouldn't it be deleted, so that it only contains the root node of the xml (the only useful information)?

      I was thinking about how to handle JSON resources. In cases like this where the useful information is contained only in the handle's ResourceExtraData, the resource cache currently doesn't account for the memory used by that extra information. Should I be overly concerned about this?

      Finally, where can I find out more about extending the cache so that it preallocates memory at initialization and hands it out when loading resources? or about caching for games more generally? A google yielded boost.pool; would that be a good solution?
    • Neurone wrote:


      In GCC's xml resource loader, the raw buffer containing the xml file is discarded. However, an empty buffer for the processed data is allocated with the same capacity as that for the raw xml.

      After the xml has been parsed, the handle still retains this empty buffer to processed data. Shouldn't it be deleted, so that it only contains the root node of the xml (the only useful information)?

      I was thinking about how to handle JSON resources. In cases like this where the useful information is contained only in the handle's ResourceExtraData, the resource cache currently doesn't account for the memory used by that extra information. Should I be overly concerned about this?

      In my own engine, I have the resource Load() function return an enum that describes what it should do with the raw buffer. There are times I want to keep the raw data and times I want to discard it. This keeps the number of extra allocations down.

      It's quite possible that we're holding on to more memory than necessary for XML resources. If that's the case, it should certainly be deleted when we're done with it. It might be a bug.


      Finally, where can I find out more about extending the cache so that it preallocates memory at initialization and hands it out when loading resources? or about caching for games more generally? A google yielded boost.pool; would that be a good solution?

      In GCC, I wrote a section about memory pools. You can see the implementation in the GCC source code. There are also some good resources on writing allocators in some of the Game Programming Gems books.

      One of the biggest improvements you can make to the resource loader is to make it run in its own thread. If you're doing any resource loading during gameplay, this is a must or you'll get spikes in the framerate.

      I tend to avoid boost personally. It's very bloated and I do my best to depend on as few 3rd party libs as possible.

      -Rez
    • Thanks, will check the memory pool resources out.

      Why would having a background thread for loading resources help? You'd still have to wait for the resource to be loaded, which is an i/o bottleneck.

      What's the advantage of having extra data attached to a ResHandle instead of creating different ResHandle subclasses for different resources e.g a JsonResHandle? I've found that this allows for more concise code like handle->getRoot().

      This is my attempt at modifying the process to support this; I wanted to handle as many different resource loading scenarios as possible, but not sure if it may be overly complicated.

      1. file is loaded from the archive into a raw buffer.
      2. if the resource needs decompression (e.g .ogg), a secondary buffer is created for it, and the decompression performed.
      3. If no decompression is needed, the rawBuffer is set as the workingBuffer. For resources that require decompression, the secondaryBuffer is set as the workingBuffer, and the rawBuffer discarded.
      4. Loader does any additional processing, returning a handle of the appropriate ResHandle class. The handle is only initialized with the resource's name and subclass-specific data, if any. E.G the pointer to a JSON root node.
      5. Cache asks the loader whether the workingBuffer should be cached. If yes, the res handle is initialized with it. Otherwise, it contains an empty buffer.
    • Neurone wrote:


      Why would having a background thread for loading resources help? You'd still have to wait for the resource to be loaded, which is an i/o bottleneck.

      For streaming. If you load a file on the main thread, the entire program haults until the file has finished loading. If it's running in a background thread, that thread will yield while waiting for the hard drive, allowing the rest of the game to keep running. You need a bit of a predictive system so you can load stuff before you need, but that's not too tricky.

      Of course, if you're doing the load-everything-you-need-for-a-level-once method, this won't help. My own system just has an interface to handle this though. I can sleep until a resource is loaded if I need to, which is what I do when my loading screen is up. I tell everything to load, then sleep the main thread until it's all been loaded. During gameplay, I let it run. I clear out the active texture between renders in debug mode so I can catch blank textures.


      What's the advantage of having extra data attached to a ResHandle instead of creating different ResHandle subclasses for different resources e.g a JsonResHandle? I've found that this allows for more concise code like handle->getRoot().

      In general, it's better to favor composition over inheritance, though I generally agree with you on this point. My own resource system has a Resource base class that all resources inherit from. Each resource is responsible for loading itself when told to by the system and consumers of the resource cast to the appropriate subclass, which contains the necessary interface for using it.

      You have a similar concept here, but it's contained in the IResourceExtraData member. Either way works and I'm not sure of any particular advantage of this method. Mike might have more insight since he wrote it, but my sense is that he just prefers this kind of pattern.


      This is my attempt at modifying the process to support this; I wanted to handle as many different resource loading scenarios as possible, but not sure if it may be overly complicated.

      1. file is loaded from the archive into a raw buffer.
      2. if the resource needs decompression (e.g .ogg), a secondary buffer is created for it, and the decompression performed.
      3. If no decompression is needed, the rawBuffer is set as the workingBuffer. For resources that require decompression, the secondaryBuffer is set as the workingBuffer, and the rawBuffer discarded.
      4. Loader does any additional processing, returning a handle of the appropriate ResHandle class. The handle is only initialized with the resource's name and subclass-specific data, if any. E.G the pointer to a JSON root node.
      5. Cache asks the loader whether the workingBuffer should be cached. If yes, the res handle is initialized with it. Otherwise, it contains an empty buffer.

      Seems okay to me, except that there are a bunch of buffers being swapped everywhere. You should make all of this as transparent as possible. For example, I have a ResourceFileInterface base class with two subclasses: ZipFileInterface and RawDirectoryInterface. My interface into loading a file from the resource system is the same regardless of it being a zip file or a directory structure. It doesn't matter.

      So the whole decompression thing is totally hidden by the interface and happens on the resource loader thread while my main thread is churning away on the game logic and my render thread is drawing stuff.

      -Rez
    • Ah, that makes sense for streaming.

      Maybe I should clarify?

      1. file is loaded from the archive into a raw buffer. // the archive could be a zip or a directory, any decompression required to get the file from the archive is done transparently.
      The decompression in step 2 is for additional decompression that needs to happen e.g a .ogg file, which is handled by the resource loader.

      I don't see any way of reducing the amount of buffers I'm allocating and swapping around. I'm using an std::vector<char> for buffers. The only optimization I could think of is to do an std::move on step 5 to move the buffer into the resource handle.