Random characters on the end of a file rawbuffer

    This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

    • Extra characters on the end of a file rawbuffer

      I've set my game up to load a particular set of files from the assets.zip file.

      These files only contain text, and have an unhandled file type, so they are opened with the default file opener.

      Unfortunately, the results aren't what I expected. As well as the expected text, I'm getting a dozen or so extra characters at the end of the buffer. (for example, half a dozen 'y's with dashes above them).

      Does anyone have any idea why this is? I havn't modified the reading process at all.

      Or am i just trying to get the results from the wrong variable?

      Thanks for any help. If you think me posting the expected output and the actual output would help, just say so. (I would now but I'm on a different PC)

      The post was edited 1 time, last by Randomman159 ().

    • This is because there is no null termination in your buffer, when you go to print this out or see it in the watch window, it keeps reading because it doesn't find it.

      The easiest thing to do is throw a null termination on the end of your buffer ie.

      Source Code

      1. char aBuffer[256];
      2. aBuffer[255] = '\0';


      Depending on how you load the file, it sometimes puts in a null termination once it has reached its end of file marker, however sometimes you will have to set the null termination yourself directly after the data in your buffer.

      If you are storing your sizes, and end up passing this to anything that needs it anyway, this doesn't do a whole lot, but if you are planning to use printing functions like printf or the cout class, you will need that null termination.
      PC - Custom Built
      CPU: 3rd Gen. Intel i7 3770 3.4Ghz
      GPU: ATI Radeon HD 7959 3GB
      RAM: 16GB

      Laptop - Alienware M17x
      CPU: 3rd Gen. Intel i7 - Ivy Bridge
      GPU: NVIDIA GeForce GTX 660M - 2GB GDDR5
      RAM: 8GB Dual Channel DDR3 @ 1600mhz
    • A better way to do this is to memset() your memory rather than just set the final character:

      Source Code

      1. char pBuffer[256];
      2. memset(pBuffer, 0, 256);


      This is safer since you may not have exactly 255 characters to read in.

      -Rez
    • Thanks for the fast replies :D

      I am using the resource handling that is part of the provided game code. This already uses memset().

      Am I misunderstanding?

      If not, what could be causing the engine to calculate the wrong size required? Could it be the use of a different .zip compressor? Or something else?

      Thanks again for the help guys :)
    • That's true memset would be the right way, my resource cache I use heap memory so my array is always the size of the data + 1, so I just ended up doing the single character on the end.
      PC - Custom Built
      CPU: 3rd Gen. Intel i7 3770 3.4Ghz
      GPU: ATI Radeon HD 7959 3GB
      RAM: 16GB

      Laptop - Alienware M17x
      CPU: 3rd Gen. Intel i7 - Ivy Bridge
      GPU: NVIDIA GeForce GTX 660M - 2GB GDDR5
      RAM: 8GB Dual Channel DDR3 @ 1600mhz

      The post was edited 1 time, last by mholley519 ().

    • two things:
      - memset is a bad idea for things like that, because you trash your cache depending on the size of the buffer (and simple stuff like that can make a big difference, read fgiesen.wordpress.com/2013/01/30/a-string-processing-rant/ )
      better:

      Source Code

      1. char temp[256];
      2. int bytesRead = ::read(..,sizeof(temp)-1,..);
      3. if(bytesRead >= 0)
      4. temp[bytesRead] = 0;
      5. else
      6. //process error


      - you have to be very careful how to read files, I've stumbled many times upon automatic line end conversion when I've used fopen with the parameter "r" which opens the file in textmode and automatically converts 0x10,0x13 to 0x10. And if you take the filesize and read that many bytes in text mode it gets wrong! (has your textfile coincidental around 16 lines?). imho it's much better to open the file with "rb"

      I've not taken a look at the source code, so forgive me if it's completly off :)

      happy coding
      questor
    • Or you could use fstream
      PC - Custom Built
      CPU: 3rd Gen. Intel i7 3770 3.4Ghz
      GPU: ATI Radeon HD 7959 3GB
      RAM: 16GB

      Laptop - Alienware M17x
      CPU: 3rd Gen. Intel i7 - Ivy Bridge
      GPU: NVIDIA GeForce GTX 660M - 2GB GDDR5
      RAM: 8GB Dual Channel DDR3 @ 1600mhz
    • Originally posted by questor/fused
      - memset is a bad idea for things like that, because you trash your cache depending on the size of the buffer


      I disagree with this philosophy. You really shouldn't be optimizing at that low of a level, especially for things like File IO. It really won't make any difference at all, especially considering that this is most likely on another thread. Safety is more important than performance in this case.

      Cache coherency is definitely good, but it's most important inside inner loops or code that's executed one or more times a frame. File loading is not done that frequently. In general, you should not worry at all about performance until you're either done with the project or the performance makes the game unplayable. That's when you should run a profiler and start handling performance issues. Most of the time, it will be higher level issues (poor algorithms, calculating or loading data multiple times, etc.) CPU caching is usually only a real issue for things that are done constantly, like looking up a component on an actor or transforming a large amount of data.


      (and simple stuff like that can make a big difference, read fgiesen.wordpress.com/2013/01/30/a-string-processing-rant/ )


      He's not talking about memset(), he's talking about optimizing horrible string processing by not doing unnecessary allocations or copies, not doing multiple file reads, and other algorithmic improvements. This is a great example of how to optimize. They noticed a very bad performance issue so he profiled it and started addressing each one piece by piece.

      Calling memset() on a newly allocated buffer should not grind your game to a halt.

      -Rez
    • I am still having issues.

      The temporary fix I mentioned in my last post is not working completely.

      Although for the existing files that were being used it was fine, for some reason as i modify the contents of a file (essentially a .txt, but named as a .frag), sometimes a few more characters are being added on.

      I'm using the method of calculating buffer size provided with the book, so that is why i'm confused.

      Could it be that the compressor i'm using (to create the .zip) is slightly different to that expected?