It took me three years, on and off, to work on my pet engine project just to display a pretty triangle.

Yeah, creating a game and your tech for the game is really hard. That's why we have a number of companies offering their engine for our game development needs.

What you're doing is more along the lines of creating a tech for game creation rather than making a game. This can be really mentally exhausting as you're only working on implementing systems after systems and algorithms and a bunch more algorithms without getting any feedback on the goal you're trying to reach until you've implement a hundred systems and fixing a few thousand bugs for your game to run.

Whatever we design and implement isn't really permanent too. So what you said is exactly right; Don't think too much on how you would want your design to be future proof and just try to implement it based on your current needs. You can always refactor it later on if your requirements has changed or needs additional features and functionality. The important thing is that every time we update a particular system it becomes better. That's how your system evolves.

I wanna tell you how i ended up taking up so long just for my pet engine to draw a single triangle on the screen. My pet project actually started about 5 years ago when i decided i wanted to make my own implementation of STL. Of course its going to be less efficient and flexible but my main goal was to learn and understand the underlying data structures, techniques, algorithms and pretty much all the technical stuff as much as i can. So it took a couple of months to implement all the basic data structures such as array, linked-list, queue, stack, map hasth-tables and so on. Obviously being humans as we are, we're bound to commit mistakes. So my data structures library was plagued with so many bugs which to this day i'm still finding subtle bugs to fix.

Then i got inspired to learn about memory manamagement. I implement a memory manager that would handle all my global data allocation needs. I bypassed the new operator so that all my calls to new would use my memory manager. I re-implemented my memory manager so many times that it took me probably about half a year, on and off, to get to a somewhat usable memory manager. Its working but i'm currently not using it since i'm too afraid it'll cause some crazy insane unimaginable bugs that i might not be able to solve at this moment. So my memory manager took about seven or eight rewrites taking about a thousand lines of code each. I also worked on a bunch of other systems but the rendering system was the most interesting part.

When i got to the point that i felt like i was ready to implement my rendering engine, using Direct3D 9 at that time, i got stuck in thinking too much how i would design my system to be perfect and still be reusable in the future. That was the time DirectX 10 was just about to come out. So i tried writing some code for the rendering engine trying to follow some open source project and half the time i actually have no idea what i was coding about. I was just literally copying everything until i realized i had no idea how i would connect all these systems for my rendering engine. So... it probably took me about 1.5 years to actually realize that in order for me to learn, i just need to do it. So i did and i ended up with a very simple rendering system that can handle rendering simple triangle messhes with a GUI system on top of it. It looked something like this:

[IMG:https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-ash4/315611_2453985279420_1721091198_n.jpg]

When i got to that point, it was the time that DirectX9 was slowly fading away and DirectX10 and 11 was starting to get some momentum... Since the architecture of DX9 and 11 are so different, I felt so depressed wasting soo much time working on my DX9 renderer that i had to scrap all of it out and create a new rendering pipeline to adopt it for the DX11. I had no choice so i did it anyway. It took me again probably about 2 years just to finally get something showing on the screen. This time i was more experienced from my previous rendering engine failure that i think i was able to create a much better and flexible one with my DX11 renderer. My simple understanding of simply creating a simple wrapper around the graphics api turned into an arguably humongous rendering system. I started out writing wrappers for each type of graphics resources then added functionality for common usage patterns, I created wrappers for each type of shaders, created a virtual pipeline that mimics the one in DirectX11. I realized that i needed a mechanism for parameters to pass through different rendering subsystems so i used Jason Zink's Flexible Parameter System. Then i learned that Microsoft is deprecating the shader effect framework so i created my own effect framework. I realized how inflexible my framework was so i followed GameDev.net's Yann Lombard's effect/shader system where it would load an effects library separate from the registered shaders then each shader would register to its designated effect if it meets the effect and system requirements. I also made a simple mesh container but because of Yann Lombard's shader system i made it so that my mesh container would be flexible enough to support different types of effects as long as it contains the vertex elements needed. So now i can just easily switch a model's shader from bumped mapped reflective surface to a simple ndotl lighting by simply assigning an effect to the object. I realized that in order for me to create, or load, more complex object i need to support a model format. I decided to create my own with the help of FBX sdk. It was confusing, and still is, to learn fbx but i managed to get the basics and designed a simple custom model file of my own. Of course i also need to write the data as a file so i had to learn xml and encapsulate it in my fstream class.

So yeah... my initial goal of a simple rendering class ended up needing to implement a bunch more complicated systems just so that i can render stuff and show it to the screen. Of course all of these wouldn't have been possible withouth the GCC book and its authors..

Lastly, would i be able to show this to my employer and get me a job? Maybe.... But more than likely they would be looking for a finished game product than some kind of an unfinished boring tech demo. But i still do it anyway. I'm interested more in how do these technologies work than the games itself. I just wish i could get a job working in the engine development team in one of the big companies in the industry someday. Or get a masters in CS in Digipen first but the tuition fee is extremely expensive especially for me living in a 3rd-world/developing country.


The end
[IMG:http://i0.kym-cdn.com/photos/images/original/000/141/644/7nTnr.png?1309330974]

@BrentChua - wow what a story. As hardware gets more capable and underlying libraries or even language features begin to include higher level code, I find there's a real struggle to learn what's going on under the hood versus just using the technology and building upon it to something even cooler. At some point I think it just becomes easier and also compelling to do the later - why bother rewriting a buggy STL library when you can spend time solving other problems?

What's really interesting about this choice is that it introduces a certain fragility in our increasingly technical world. At some point basic understanding begins to fade, and engineers rely on expensive and hard to find specialists to solve serious problems - as those people become scarce you can find yourself in a real pickle. I was on a project a few years ago that needed some serious PS2 hardware knowledge, and it simply wasn't around - for any price.

Core understanding is great - but seeking a balance between what you want to make for yourself and what you can grab off the shelf is really important. I can understand how an electric drill works without building one myself - and if I just go grab one from a store I can spend my time using this tool to build some amazing things. The same is true for programming.

It makes sense to learn enough of what's going on at the bottom level - especially the hardware level - so that when things go wrong you can look at the entire problem and find solutions. It's easy to skip past this understanding and just use the tools provided to you - your project progresses quickly and everything seems fine. Until of course, something goes wrong. Without some understanding of the lower levels you are forced to try random solutions. Random solutions on something as complex as a game will get you nowhere fast.

This, I think, is one difference between becoming a professional software engineer and remaining a hobbyist. Both are commendable and valuable uses of a person's time of course, but one path creates a career while the other a pasttime. The professional will always do a deeper dive, but have an intuition when to stop and come back up for air.

Again, what a great story - thanks for posting it.