by Johannes 'johno' Norneby
contact@nornware.com
This is a transcription of a monologue that I did in April of 2017. I've kept a lot of the 'stream of consciousness' style intact in order to retain the feel of the original talk. I hope it's intelligible, entertaining, and informative.
I guess I could talk a bit about certain fun things that have happened to my programming style during the course of Early Access Development of Space Beast Terror Fright.
Some people know this, I've written and talked a bit about this, but I've been finding that I'm really moving away from Object Oriented Programming, which is sort of funny because when I was starting out in the games industry in the late 90s I was really a super-duper fanboy of the whole idea of higher level programming. Of course I wanted to do games because I was really affected by the original Doom (1993), and that's what really got me into being interested in game development in general. So I wanted to do stuff that was like that; real time graphics and high performance stuff.
When I started programming seriously it was really C++ that was all the rage, that was 'how you should be programming', and Object Orientation was a huge part of that. Being exposed to C++ you learn initially that this is something that was built on top of C, and C is procedural and does not have the object oriented things. It seemed (you know, being young and everything...) like it was just so obvious that "Yes this is better! This is an evolution of programming! This is where the state of the art is at!
And I guess in a lot of ways that maybe I've had a little too much trust in the authority figures in my life, I would have to admit. So I was thinking: "Ok, the people who know stuff about programming, they say that this is what you should do, so we'll do that!" And to be honest, that's what the whole industry seems to have done since then; Object Oriented Programming is... what are you gonna say? That's what we do!
So it's interesting to find 20-something years later, after have been programming seriously all that time, how I'm really really just full circle moving away from all of those things that make Object Orientated Programming 'better' or 'different' from procedural programming. For interesting reasons. For reasons that come from not any sort of dogmatic or philosophical place but really from actual practical experience working on game development on a tiny budget, trying to get some things to work that are motivated by "Oh, I want my game to do this, what is the best way?", the most efficient way to use my time and you know all of that stuff. Maintainability of course is a huge part of everything.
There's some background to it...
I've been working on my nornware codebase, which has always sort of been the hobby codebase outside of any serious job, for a really long time. I started to realize, wow, I've been working on this in some way shape or form since really 1994 or something like that. There's always been this aggregate thing on my hard drive that was evolving and growing. I guess all the indie development stuff came out of that in parallell to working at primarily Massive and of course experience from DICE and some other places too.
But it was really I think... let's see, let's get this right... I was Object Oriented all the way until maybe I started working at Dreamler from 2013 - 2015; I eventually quit that job to seriously work on SBTF full time. My job at Dreamler was close to game programming because the idea was to build a planning tool that required real-time graphics and real-time networking and things like that, so they hired me based on my experience with real-time systems (like games). It was really a lot of custom graphics programming and low level network stuff.
What happened was that I came into contact with two different styles of programming, which turned out to be a good challenge for me. I basically wrote the basis of the client codebase and I was really still into serious Object Oriented style; I based it off of what I usually did.
But a guy came in who was much younger than me, just out of university, and his style was very very much modern C++11. He was into a lot of new C++ features that I hadn't really cared to become aware of, a lot of lambdas and things like that. It was funny because I found, wow, I didn't really like things like that! I'd had some previous experience with things like Boost, the whole philosophy behind things like Boost, which I interpret as "Oh, we want C++ to both be super high performance and low level but we also want it to have all the modern language features that you would find in something like Java or C# or something like that; we want it to be both low level and high level."
I'd found that maybe perhaps, even though I was thinking that I was doing Object Oriented Programming, I guess I found during the years at Massive and especially after Massive that maybe Massive didn't really go all the way into all of that stuff as compared to some others who would do lots of template metaprogramming and stuff like that, use Boost and all of that. So I guess I was maybe a bit conservative to begin with, and when I met this guy at Dreamler it was like "Yes, I seem to be very conservative! I'm Object Oriented but I'm not super into the state of the art...", I suppose you could say.
And then another guy joined the company and who was another style of coder. He'd been doing a lot of procedural programming I think in PHP, and he wasn't really used to ANY form of Object Orientation, so he even felt that my stuff was way too object oriented, not to mention the younger guy coming on. It's like "Oh!", so this sort of interesting discussion started up like "What is a good style?"
I mean I guess that maybe for the first time since leaving Massive and DICE I'm talking to other coders about "Where should you be at? What's effective? What is efficient in small teams?" and all that stuff.
The procedural guy, I guess he really started affecting me a lot; it's like "All of this Object Oriented stuff sort of feels like boilerplate, why do we need all this stuff? What's the point of all of this?". So I guess during my time there I sort of 'devolved further', fell further away from wanting to do any sort of abstract programming, any sort of polymorphic crazy super elegant high level stuff, even more than before. So that was interesting, and I found that while working on nornware stuff on the side that that style started infecting my own codebase too.
But the real kicker was working on Space Beast Terror Fright...
Eventually I quit Dreamler, working by myself again as the sole programmer (with artists), and I was just so under the gun. The interest for SBTF happened sort of out of the blue, it was nothing I was really expecting, and I really felt that "Oh I gotta get on this! I have to quit my day job, I have to work on SBTF now because it's happening NOW and I really wanna roll with it now that people are interested!". So the whole push of getting it out into Early Access as soon as possible after the demo, about which in some ways I suppose I felt that maybe I'd given too much away perhaps. It's great that there was interest for it, but the whole initial rush was "Oh my god, have to get SOME other feature in there that would make it ok to charge money for it!". That feature ended up being local split-screen co-op, with XBOX controllers.
So I was even more than before in a situation where you have to be effective, you have to be efficient, you can't spend time writing code that, you know, sort of doesn't do anything but looks nice?! So the whole move into minimalism was sort of to a further extent reinforced by the state of things.
SBTF got to the point where there was networked multiplayer, which was a very important feature of course. We rolled out a version that we had of course tried on a local network, played across Gothenburg, and also played across Sweden, trying to break it, trying to find out if the architecture worked and if it was something that we could ship. The funny thing, such a first-world problem, is that the internet infrastructure in Sweden is just so good, so that the initial implementation which was a LAN-style peer-to-peer architecture (peer1), it worked, it just totally worked for internet play in Sweden.
So I was, ok, always with the interest of getting updates out there as often as possible, I had no real expectation that it would work internationally. This is not tolerant to high latencies, because technically what I'd done was introduce just a little bit of input buffering (4 ticks at 60Hz), which of course adds latency that you can feel as compared to singleplayer. Any situation where all nodes couldn't get those 4 ticks of input across the wire in time would simply block / the simulation would stall and it was just horrible.
And of course this was the feedback that we were getting in the forums back from players; it doesn't work, it's laggy, it's terrible and we want it NOT to be, right?! That was the challenge there, with people saying things like "You know I really want to play this internationally with my friends, like I'm in Europe and they're in the States or something and I wanna play it!"
In some instance someone said "I'd rather play this than CS-GO, so please fix it." And I was like "I know that that's really hard with this architecture, but oh my god it's just such a challenge because if people would rather play this than CS-GO... that's awesome!" It's awesome that people care, and the space that this game is moving around in is sort of the same as these really high production value first person shooters, and that was just way too good to be true.
So I was both consciously and unconsciously thinking about what could be done to solve the problem of making SBTF able to be played at higher latencies, the typical big woolly wild internet.
While working on other things, and not sleeping much I suppose, I sort of got this idea into my head of just naively looking at the problem; what is the thing that people do not appreciate about peer1, what is the problem? The problem is that the simulation stalls, and the reason it does that is because there's just not enough enough time to run the game at 60Hz and have all the inputs arrive for the given tick that you need to simulate in what is essentially 16 milliseconds.
It's kind of frustrating and stupid, because I know what my local player needs to do because I'm generating that input locally and I have that input and I could tick (simulate) for myself. But if I do tick for myself then all the inputs I need for everyone else are missing, so if I tick I'm just cheating, it's not the truth.
The whole idea is based on determinism, it's like a chess game I guess that's going really fast. To do remote simulation that everyone needs to do on their own machine; it's like:
peer0 -> peer1 : "I did this for tick 0, what did you do?" peer1 -> peer0 : "Oh I did this!" peer0 : "Ok, so I know what he did, and I know what I did myself, so I can do one step..." peer1 : "Me too!"
and then you keep doing that and it doesn't diverge because of determinism in the codebase that is running at a fixed tick rate and all that stuff. I mean that's a whole other issue, if you don't understand how deterministic input driven networking works, of course you need to go look at that...
But anyway, it doesn't work at high latencies. It's not possible (in the time available) to get the information sent between all the different machines for that to work. So what is the problem? The problem is the stalling.
It's funny, being a game developer for a long time, it's obvious in network games, and I played a lot of Quake and stuff like that back in the day, it's like; with the client-server architecture, one of the things that's sort of seen as a plus, and again, SBTF is not client server, but client-server is sort of like what the Dooms and the Quakes evolved into. id Software and John Carmack were really the pioneers of this, you know sort of figuring out that it needs to be client-server. There are a lot of things there; the server can be a beefy machine, the server can be on some good network, but mainly I think that the take-away for the user is "If I have a good connection, then the game will be good for me, and I don't care about anybody else. I only care about my experience." And that can't be faulted, that's just human psychology, that's how people play games.
And again that's reinforced by the fact that most if not every FPS that's out there uses a client-server architecture; of course I can't know that for sure but it seems to be the trend. Definitely it was the case for SBTF that the feedback in the forums was along the lines of "Oh, it's peer-to-peer, that's the problem, that's why it's bad and you need to switch to client-server!" And in a sense you could say that's true, because it is known since the 90s that you can't do peer-to-peer on anything other than a LAN.
And of course I knew this, and I've done a lot of network programming (at Massive I did a lot of network programming), and we always ran the Ground Control games on client-server, which was an interesting thing in and of itself because that wasn't supposed to be possible, with drop-in support and moving lots of state across the network and everything (the thing we kept hearing was that Real Time Strategy games couldn't be done client-server).
So I knew all about the trade-offs and that was really, without even thinking about it, why SBTF was peer-to-peer to begin with; "Oh, we're in Early Access, we need to get the networking working QUICKLY because we want to sustain player interest in the game, we want things to move forward!" So one of the nice things about peer-to-peer given the fact that the game already was deterministic is that it's a relatively small thing to add, and it's really separate from the whole codebase. If input driven determinism is your solution, everything that works in single player will automatically work in multiplayer, and that's really important because it means that there are fewer special cases. It's about economy, it's about a single programmer trying do something that's really way too ambitious on no budget, so you have to make those choices, and that's why peer-to-peer was the shortest distance to some sort of networking functionality.
So now we're in the situation where peer-to-peer was there and it's 'solid' but it doesn't work at high latencies, so how do you solve that? And it seems initially like you can't solve that because there isn't enough time; you need the inputs from everyone else NOW, not LATER, and that's what causes the stalling and that leads to the user experience problem; "I don't want it to stall! I want to have a good experience because I know that I'm on a good network and I don't really care about anyone else."
So looking at it from a sort of naive brute force angle; we can't block! If the game DID NOT block, then everything would be really good, because all the information required to do the simulation, aside from player inputs, is really available, because the networked mode and the single player mode are exactly the same thing, the same simulation. The only problem is that there are external things, which are the other players' inputs, which need to come across the wire, and if they are late; that's the problem, because the whole idea is to do remote simulation on each node (each player's machine) that does not diverge, because otherwise people won't be playing the same game, right?
And of course, there is a lot of scariness involved in doing things around determinism. It's always sort if icky when I'm messing around with the codebase; is it possible to do things that break determinism? Nobody is going to tell you explicitly if you do break determinism, it's not like the code will crash or not compile or anything, but rather be very subtle and you won't necessarily notice it. One of the ideas that's always there is: how do you technically validate determinism? At this point there was no such technical validation of determinism to answer the question: does the simulation diverge? We just went ahead with "It seems to not diverge!" so that's what we shipped.
But back to the question of how to solve the blocking / stalling issue, without making the architecture more complicated, without having to give things up; I didn't want to give up the fact that the singleplayer and the multiplayer 'just work', they're the same thing. I didn't want to move to client-server necessarily, because my sense of the thing was that there is way too much state to move in client-server. And that needs to be talked about...
My experience at Massive with the Ground Control (RTS) games, those network architectures were sort of messy because they were a mix of reliable and unreliable networking protocols. For each client to the server we had a TCP/IP connection for the reliable things, I think we did stuff like map loading and all the state that really needs to arrive in-order and not be dropped, and even things like player commands that we deemed to be requiring of reliable delivery. And then all the incremental state flooding from the server to the client was all UDP, so things like positions of all the units. We knew that that the only thing that was of interest to the client was only the latest data, so it was all designed to be robust for packet drops; we don't do resends, we don't care, just give me the latest state and I'll just apply that.
I'd always been really interested in network programming, but it always felt really icky what with the mix of TCP and UDP, the explosion of all the sockets you need for that, and the thing that we obviously saw in all the Ground Control games was if you had a reliable server to client command that says: "Create a new tank!", the client gets that and it's all good, and then the server starts spamming unreliable updates to the client, here are the position updates for this thing. Then another reliable command comes that says: "Destroy this tank!", so remove this object. Then you're always potentially in the situation where you get updates a little bit later which are also potentially out of order, that say to update that tank that is now destroyed. So you always had to at an application level say "Oh that's ok, ignore that, that's by design...", as we knew that that can happen, so you have to be robust to trying to update something that's already been removed.
So philosophically it's really strange, it's a messy architecture. Also can be noted is that when I originally wrote the abstractions over actually writing data to the wire I made an erroneous assumption; the game had a concept of a Message to which you could write various datatypes to a byte-buffer and then send that across the wire. But that was always viewed from an application point of view as a cohesive thing, and when I send this across the wire it's going to be delivered cohesively as well. Of course you know that with TCP it will arrive, as that is enforced by the protocol with resends or whatever. With UDP you know it could be dropped.
What's funny is that both those things are true, but UDP is a datagram protocol; if you get something you're going to get the entire cohesive datagram, as long as you are under the MTU size. The misconception was that if I send the same thing over TCP I'm going to get the same cohesive thing in a single go, which is not actually true, because TCP is a streaming protocol, so there's no concept of a packet there. There are of course technically packets under the hood, but that's not what the protocol is exposing to the client, it's a stream. So you send 500 bytes, you may NOT get those 500 bytes in a single cohesive recieve call.
So that really messed (also) with us until we realized that we had to take that into account, and if you are imposing some kind of application level structure or message you have to understand the differences between the protocols. So that was also another source of ickiness; that's not fun, that's not good, that's not smart.
So in the years after Massive working on other stuff, not necessarily network stuff, I read a write-up by Brian Hook on what John Carmack had done on Quake 3, and it was sort of pitched as "This is when we really got it right." The whole idea is really super simple from a protocol standpoint; the client has an upstream packet that fits in an MTU, keyflags and view angles for a first person shooter. The server is only interested in the latest version I guess for the real time stuff; there was some stuff in the article about how to do stuff like chat messages that are supposed to be reliable; you do it brute-force by having a field in your packet that is the chat string, and you keep spamming the same chat string until the server detects a new one, so basically just piggy back stuff on top of the basic unreliable stuff until it brute-force arrives, which is interesting. But the whole point is that you get rid of the need to have TCP/IP in there, you just have UDP and you do all the stuff you have to do anyway on the application level to be resilient to packet loss and out of order stuff.
But even more interesting was the blatant idea that the server just continually sends packets describing the entire server state to the client; here's the state, here's the state, here's the state... so it's not a message / event based, very verbose protocol. But you think, well that's impossible, you would be very limited in the kinds of games you could do because if your MTU is 1500 bytes that limits the amount of gamestate you can have. Quake 3 does advanced delta compression (as well as splitting compressed state into multi-MTU representations) to get around that, read up on it.
But regardless, the whole idea of having just UDP and also having a simple protocol where the client says one thing and the server says another and that's IT, that was really different from what I had done earlier with the Ground Control games. And of course in the interest of having things be super duper efficient and making games on a super small budget, all of those sort of simple and small solutions were always of interest.
So definitely for the original (peer1) version of SBTF it was really that sort of thing, the client has one kind of packet, key/button flags and view angles and that's IT, and that's easy to get into an MTU; the interaction fidelity isn't huge in a game like SBTF. And of course the server state compression problem that Quake 3 had I do not have, because what is happening is that all players are sending their inputs to each other, there is no server, a star topology.
Each node sends speculatively:
peer 0 -> peer 1 : "Hero are my inputs for ticks 0 thru 3, I want tick 0 and onward from you." peer 1 -> peer 0 : "Hero are my inputs for ticks 0 thru 3, I want tick 0 and onward from you." peer 0 -> peer 1 : "Hero are my inputs for ticks 4 thru 7, I want tick 4 and onward from you." peer 1 -> peer 0 : "Hero are my inputs for ticks 4 thru 7, I want tick 4 and onward from you." etc...
When you get an incoming packet you just store that as official input from that other player for whatever tick they said. Also you note "Oh you asked me for tick 0, I'll send you 0 and onward..." So it's really simple, and through that kind of acking it guarantees eventual delivery of all the inputs to everyone.
So it's a really simple protocol and fantastic that it can work, but it's not latency tolerant. So moving forward from that; what could I do to make it latency tolerant and not radically change the architecture. I didn't want to change anything outside the networking layer and see if I could get a way with that.
So again back to the problem; the stalling. The game needs to simulate NOW in order to keep up with the 60Hz simulation rate in real time. The realization is that for the local player I can always simulate correctly because I'm the one who is deciding what that player does, the inputs; the peers are authoritative about what they themselves are doing with their players, I know what I'm gonna do. That means that you get a single player sort of experience without any lag or stuttering for your own self, because you're decided where you're looking and what you're pressing, and you're running the simulation locally. So the naive thing is; just don't block! Run the simulation locally and just go!
So the question is; what inputs do I use for the other players? If I'm on tick 67 I can generate the input for myself, and I'm just not going to have the other players' inputs. So the philosophical thing was; what would happen if I just went ahead and simulated and just said, ok, I'm obviously going to use the latest local input for my player, but for everyone else I'll just assume that they're still doing what they did for the last simulation tick; I'll use the last received input from them. And that's really what happens in client-server FPS games, because that way the server can keep going; I'm (the server) going to use the last thing that you sent to me, I'll just buffer the last thing that I got from each client and I'll use that, that's 'what they're doing', officially, until they till me (the server) otherwise.
So that works the same way in SBTF, but the problem is that since there is no state replication and there is no node that is the 'server' that tells everyone what is 'officially' happening, what's automatically going to happen is that the players' experiences are going to diverge, because everyone is going to do the simulation for a given tick in a different way. Different players will have different pieces of the whole puzzle. So you get a smooth game experience, it's never going to stall, it's going to be like single player, it's going to be awesome, but it's going to diverge, so obviously you break everything, it's not going to work, it's not multiplayer, it's not a shared simulation anymore.
So I was thinking; that's going to solve the stalling but it's going to break the game, so it doesn't count, not good, but it was still interesting because I then started thinking: What does it mean 'to tick the simulation'? It's sort of a small mutation of the game, because running at 60Hz the game doesn't change that much from one tick to the next. I started thinking; what would happen if you isolated everything in the simulation that potentially COULD change for any given tick? That led to the realization that there are obviously some things in the conceptual 'server' (or as I call it: the model), the invisible purely logical part of the game which is the actual logic, that has nothing do with fancy explosions or graphics or anything like that; you know coming from a client-server background, I'd always put up a very strict barrier between "This is visualization stuff and this is simulation stuff, or pure logic..."; that was a core aspect of all of this, something that I guess could be talked about at great length, the whole idea of going with a sort of Model-View-Controller idea where the visualization is separate from the actual simulation.
So I had this library in the project called model which contained the simulation, so it was very easy to reason about what was being mutated by a tick, a rather small piece of the entire codebase. But even further beyond that I started realizing that there were parts of the model which are immutable for the duration of the game. Things that fall into that category are things like the collision for the walls, the pathmap, the locations of datacores, and there are so many things that are not going to change, and these are things that can be effectively moved out of the concept of a mutable simulation.
So in the pursuit of all of this I did a big split; what in the model is immutable and what is mutable? You sort of start seeing that there is a of lot state there, a lot of space beasts and a lot of moving parts, but it's conceptually very clear; the heros move around, the astro-creeps move around, the space beasts move around, the doors open and close, the datacores go from active to inactive, a lot of things like that. But the thing that was in the back of mind was (and some of you may already understand where I'm going with this); how much state is that? And can that state be really effectively isolated and packaged in a way so you can reason about EXACTLY what changes during a given tick? Again, the premise is that we're going to be doing SPECULATIVE simulation of the game in a divergent way, but MAYBE it might be possible to do fixup on that later.
So what I ended up doing, and considering the implication of the title of this article, why I gave up on Object Orientation; it was because of this. I was looking at all of this stuff, the structure of the code, and realizing to be able to boil this down to exactly what changes for a tick, I was thinking that I needed to look at the data separate from all the other infrastructure. And when you're doing OOP there's always; here's an object, here's a class, it has methods and it has state, but I really started feeling like I really wanted to isolate the state. And I knew you can do procedural programming so you can have like 'a struct of structs' which is just state and then you can have functions outside of that which operate on the state, which is supposedly all ugly and public and not encapsulated and everything, but I was thinking maybe that's interesting.
So what I did was a big refactoring, partially to separate the immutable stuff, and that stuff wasn't necessarily (initially) moved to procedural style, but then I took all the mutable stuff and pulled out all the data and made that into a single big blob; here is the entire mutable state for the entire simulation, everything was refactored to be just functions operating on that blob of state, and got that to run and everything's exactly the same and there's no change in functionality, there's no regression, nothing like that. So it still works, all the networking still works, but it's interesting because you start seeing that this thing is just a single "buffer of memory".
And one of the things that I had to do there was make hard decisions related to that the state needed to be of a certain size, not necessarily any specific size, but it needed to be a fixed size; I didn't want to have any dynamic memory allocation, any growing arrays, anything that was structurally complex, so I made hard decisions about how many space beasts maximum, how many astro-creeps maximum, how many bullets maximum, how many datacores maximum, all that stuff was just hard-limited, and of course I did lots of tests to see what were reasonable numbers, but then I just made a decision. The size of the mutable state is always the same:
struct mutable_t
{
beast_pool_t beasts;
creep_pool_t creeps;
bolt_pool_t bolts;
mu_door_pool_t doors;
mu_sentry_pool_t sentries;
mu_core_pool_t cores;
mu_fence_pool_t fences;
breach_pool_t breaches;
scientist_pool_t scientists;
team_t team;
mu_panic_pool_t panics;
rules_t rules;
core::xorshift128_t prng;
tick_t tick;
};
So now there's a big struct, basically a big blob of bytes, say 40 kilobytes or so, and that's the mutable state. That's what changes during a tick. That's ALL that changes. It became so clear to look at it that way; you run the function called tick(), passing a writable reference / pointer to the mutable state, constant references to the input for that tick, etc:
void tick(const tick_input_t& input, const immutable_t& im, mutable_t& mu, tick_output_t& output);
It's like grinding the gears a single step, I visualize it sort of like a self-playing piano. You start looking at what a game simulation actually is, and you're like, there's no magic there, it's just you run this function and you mutate this blob of data one step. Because again, in the back of my mind I'm thinking; I'm going to be diverging by allowing all the peers to just go and never stall, but I'm sort of thinking this evil thought; what happens if I can do BACKUPS, right?
So I'm thinking when I'm running the network version and I'm ready to run say tick 50, I always have my local input, but I don't have any of the other inputs, so I'll say; okay, this is not an official tick. I'm going to diverge from the official simulation. Fine, I'll do that, and that's really what I'm basing all the visualization of off, this potentially dirty, potentially divergent version of the mutable state, the simulation.
But what I also recall is that when I'm going to do maybe tick 51, or some other tick in the future, perhaps it could be the case, potentially, that I do have all of the inputs for tick I want to do (which was the original peer1 case because of some buffering). Then I can say, okay, this is an official tick, I will mutate the official simulation, and then I'll back the result of that up saying "Ok tick 51 was an official mutation of the simulation", because I had all the correct inputs. And you might see where this is going...
Obviously initially you're going to be doing a lot of speculation (unofficial ticks), but at some time later you're gonna get across the network all of the tick inputs that you SHOULD have applied, and then you're in a situation where, oh, I can revert back to the last official mutable state (which might be nothing / the initial state), overwrite the mutable state with that backup, apply the correct inputs, and then back THAT up as the current official state. Then to get back to where you were is you need to resimulate / re-predict back to where had previously predicted to, based on the new official state as a foundation.
So it becomes this conceptually weird thing where you're doing some speculative simulation, but when you get 'real stuff' from the network, you turn back time, do some REAL work, and then fast forward time back to where you were, because that was really what I ended up as the syncing mechanism; how does anybody know what the current time is, what tick should we be at? I piggybacked that as well in the client packets, everyone says "I have speculatively predicted to tick (say) 100...", so everyone runs to predict up to whoever is first, and that is the sync mechanism. For me, I can't motivate this in any way shape or form theoretically, but it seemed to be the right thing to do, and that's generally how I work, I try stuff in code, and I'm not good at that sort of super logical analysis, it's more of a gut feeling thing.
So this is basically what I implemented, and what I realized is that suddenly you're doing potentially many many ticks of the simulation, either official simulation or re-simulation, or prediction or whatever, but that has to be really fast, because instead of doing one tick every 16 milliseconds you might be doing lots of them, depending on variations in latency and who's slow, etc. It's still peer to peer, we're limited by the person who is the slowest and the furthest behind, and that persons is also going to have to do lots of re-simulation to catch up to the other players who are the furthest ahead.
So obviously you need to optimize stuff, this simulation code needs to be SO fast that you can afford to do it. This game is still single threaded, running at 60Hz, you need to be able to do this stuff without exceeding your time budget, because otherwise you start dropping frames and you start affect the whole networking thing even more adversely because you're being the slow person, right? So it's horrible that way, why would you even try to do that? But it's still nice because the whole idea is still completely isolated from the rest of the codebase, still an external thing where you're going in and fixing up the mutable state of the simulation in some magical fashion, and nothing else in the code notices this.
Initially the performance wasn't there. It was partially a matter of refactoring to the fixed size mutable state, cleaning things up to be as simple as possible. I was just assuming that I needed to be able to do:
mutable_t speculative; mutable_t official; official = speculative; //"atomic" operation
Copy a whole block of memory at once, that was the idea; I didn't want to do deep copies of objects graphs, dynamic allocation, etc, I just wanted to copy byte buffers back and forth; there's the official state and the speculative / dirty state, two versions of the mutable_t that are ping-ponged back and forth; so I really wanted the copy to be fast and simple, and that's really where the procedural idea came from, there's a struct of data and it's a fixed size and all of that.
So that was the one part of it, but I had to write a super low level CPU-clock timing utility to look at exactly what is expensive in the simulation, and I did a super vigilant job of optimizing the crap out of everything. In some parts of the code, there are look up structures, a lot of magic things going on to be able to avoid linear searches, in general super low level performance stuff that of course all helps the game in general across the board; it's a better game, it's more optimized, you can put more stuff in, it's good to have the headroom.
Eventually that was all done, and it seemed to work, at least locally on a LAN. It was a horrible situation because, again, peer1 always worked for us, even across the country, so it's like; how are we testing this? It's terrible, you're doing experimental stuff, it SEEMS like it should work, but I hadn't sat down and really worked out the implications of "What is this all actually doing?!", hard to visualize, what is the actual rate of ping-pong between official vs speculative simulation, etc. I did some raw visualization of it all, but it wasn't really helping us. So it was like, oh my god, this is completely experimental, it seems to work on our end, and I knew also that all the original input latency that peer1 had (the 4 ticks), which I noticed locally because single player and networked games had a slight difference in feel to them, one of the good things about peer2 is that that stuff is completely gone, you're always running what feels just like a single player experience, always speculating, just going. It seemed like if we would be able to offer an experience that got rid of all of that self-imposed latency, then it might be worth all of the other crazy stuff we have to do.
So we put it out there, and it was super scary, but the gist of the feedback from everyone was "Thanks, now we can roll!" and "Thank you very much, you fixed it!" And then it just... continued to work. And that was just horrifying in a way. It was wonderful that it did SEEM to work, but again there was no validation that it did NOT diverge, no checking or hashing or any kind of feedback loop in the protocol that says 'this is actual working'. And sure we got some reports where people were like "Oh, I had a situation where people were bouncing around and it was lagging and everything broke...", and of course that seemed to be logical that you're probably gonna end up in a situation where when someone is really behind, everyone else is doing so much work with the re-simulation that eventually there's gonna be a tipping point where it just explodes. And to this day that still hasn't been explored fully.
But in general it seems like this crazy idea allowed it to be more latency-tolerant, and supplied a reasonable experience to people who were playing the game internationally. The first time someone said "Oh yeah, we tried it between Texas and New Zeeland and it worked, good job..." I was like, ok, on to the next thing, because there's really never been enough time in this project to actually follow things up properly or dedicate people to making sure this thing is really solid or whatever; on to the next feature. But I guess it's a really happy story because it's one of those situations where my intuitive sense of what MIGHT be possible sorted itself out by simply 'typing stuff', trying stuff out and, you know, in a sense being really fearful of the implications of all of this but sort of slowly moving forward step by step and it sort of worked out.
Apparently this kind of stuff is called 'eventual consistency' and it seems to be a hot research topic and a really hard problem, so I guess I can pat myself on the back for having solved it and being in that space at all, it makes it interesting. But aside from all of that craziness and all of the good stuff when it comes to "Oh, SBTF is networked in a way that makes it more robust...", that was really the beginning of my 'official' break from Object Oriented style to procedural style.
SBTF Peer2 only (initially) changed a subset of the model, a subset of the simulation that was now in a procedural style with functions and structs, and I really really liked the way that that code looked. In a lot of ways it became smaller and cleaner and it seemed minimalistic, because the only thing that was there was the data that was required, really cooked down and just the functions, and I saw that a lot of things that you typically have as private member functions in classes, those method prototypes moved away from the .h files into the .cpp files.
What happens is that private methods in a class header become file local (static) functions in the .cpp, so you start realizing from a dependency standpoint that wow, this is much slicker because now the public interface of anything you do (.h files) is ACTUALLY only the public stuff; that's always been a thing with C++, if you look at a class you may have a very small public interface but then you have tons of stuff that is private. From a compiler standpoint, and this has been said before by others, it's kind of irritating that changing a 'private thing' in a class still causes all includees of that header to recompile. And that is a thing that changes when you do procedural style, because things that are 'implementation-wise private' become ACTUALLY private.
This really has nothing to do with encapsulation of state, of data, because obviously if you have a struct and free functions that operate on that, all struct members have to be public, so it has nothing to do with that. But I sort of got the feeling that, wow, this is actually much more minimal, you don't have to do a lot of function prototyping (as you need to with method signatures in headers), there's less boilerplate, and that style for those reasons is just sort of 'infecting' the rest of my codebase.
And since my codebase is really old, and there are about a million lines of code there, it's not to this date really the case that everything has become procedural, but it is the case that when I touch some system I will try to change it to structs and functions; just because that is really my feeling now, it seems to be simpler, more efficient, less noisy. There are cases where I still do interface-style polymorphism, the Java-interface thing, modern language style thing, where it's really just a virtual function table, and that's interesting because in those cases it's really abstraction, I don't want to see any state, I 'really mean it' as Rich Hickey would say. That stuff is still syntactically, I suppose, nicer to do with C++ inheritance, because otherwise you'd have a struct of function pointers, which is the C-style. They're technically equivalent, but honestly I'm probably bad at the syntax. So there are cases where I still technically 'do inheritance'.
I'm really really liking a lot of the feeling of the fact that there is no more this pointer. And I think for me that took a long time to get used to, but when you look at it, just do a straight port of something, you end up in all the functions that were previously methods, you're going to have an extra argument; you can't call it this, but you could call it self or instance or something like that, and you pass that along at the front or the end of the argument list. It's usually mutable (not const), because if you think about it; anything that wasn't a const method previously you need the mutable self passed along. For a const method, you pass a const reference of course.
But it was interesting to see that the self could sort of get lost among all the other arguments to the functions, especially if you're not consistent in your ordering, and then you're feeling like "Well, what's special about that argument? What's special about the self?", because I think that when you're doing OOP, the object itself is sort of the first class citizen, and that sort of stems from the what encapsulation is about, this object operates on it's own data, it's own private data, and "I'm the only guy who gets to touch my private parts, right? Or my friends..."
But but now when it's all public you realize; this is just an algorithm that's operating on some state. And you really get that feeling of; here's the state, and over here are algorithms that operate on that state. And then when you use that style for a long time you start thinking about; where does this 'only methods touch the state' idea come from? I guess it's a divide and conquer thing, I came to realize, that one the philosphical ideas behind OOP is that you divide all of the 'massive amount of state you need in your program' into little boxes, and then you associate the operations on that state into classes, and that becomes the higher level construct in and of itself. That's where Managers come from, and Systems and all that stuff. You have to 'ask the guy who owns the thing to play with the stuff', wherein procedural style there's state and you're messing with it directly, and it becomes sort of a shared responsibility thing across the entire program.
But the point of my peer2 refactoring was that I needed to isolate ALL the state in order to be able to do operations on that state en masse, right, I need to back up ALL of the state, but of course when you're running a function on a space beast you're only touching a little piece of that state. But my point is what happens is that a lot of functions that were previously 'in a class'' start moving around, and it's no longer the case where you're feeling like for every concept (like a space beast) I need a header file and an implementation file which you would do in OOP C++ really. In Java it's really the rule; one class per file or whatever, but in procedural style that stuff just explodes (with many structs in a header, functions that operate on a given struct in different translation units, etc) and I find that there's no real reason to say that 'one concept per file'; it's just a a module, it's just a translation unit, that stuff becomes very arbitrary.
And then if you continue to look at it that way you find that "Hey here's a function that calculates something based on the state of a space beast...", and you look at "Hey, where is this function called? Oh, from some part of the visualization, but not from anywhere else...", and you think so why is this the 'responsibility of the space beast' (which is part of the logical simulation)? Why does the space beast 'have a function' in the model / logic library that only the visualization cares about? You're realizing that this is a data transformation, some algorithm on the data that has NOTHING to do with the simulation. And then you start realizing that that kind of thinking is a caveat of the whole "The space beast's state belongs to the space beast!", so if you want to know something about the state of the space beast, then you have to ask the space beast, instead of "Oh, I'm a visualization guy and I can look at the space beast MYSELF and do something based on the state, and the space beast DOESN'T HAVE TO KNOW!"
I guess some people are going to scream "Everyone is touching your private parts!", but again, it becomes philosophically interesting because all of a sudden 'state is just state', and transformations of state or calculations based on state become very very separatable and they become very minimalistic. In general I feel that the codebase is just much tighter and much more cohesive and to the point, with less code total.
So it's very interesting how things just sort of, in some ways, got mushy; there are no really strict rules about "Where does the code go? Where are the functions?"; that stuff, I'm finding, really doesn't matter in a procedural style, whereas in order to have to deal with the public access to everything and having everything conceptually accessible; I'm not talking about global state, but it's still all public; is that you really start thinking about WHAT state you have, and that I think is really the big win.
Conversely, if OOP is about "Protecting yourself from messing around with all of the state you have in an uncontrollable way"; that seems like the big win, the whole point of it, but what I've found is that when you're doing divide-and-conquer OOP you end up having so much state that you 'don't even know about', it's like you're once removed from the actual state and you're sort of 'playing like there was no state'. Whereas in the procedural style you're very very very aware of what the state is, and what the MINIMAL STATE COULD BE in order to do any certain thing, and I guess that instrumental to the whole peer2 thing was minimizing state and making everything super super tight.
But I'm really finding, and I supposed I've known this from early on, that all of the problems that you're going to have in a program are based on things that relate to state. There are trends in functional programming about idempotency and all that kind of stuff; those pure functions are good because they don't have any side-effects; your bug isn't there, that's the thing. But if you have a lot of state and especially if you have a lot of caches and a lot of duplication of state, that's where the complexity of programming really really is. So if this style forces you to be more vigilant BECAUSE everything is publically accessible, again not global but publically accessible once you have a reference to something, it will make you write tighter stuff.
And to be clear about the globality of things; people might say that procedural style gives you global access to a lot of mutable state; that in itself is not implicitly true, you can still use a style like I'm doing where all of the access to state passes through the call graph. For any function to operate you have to actually pass all the stuff into it, and that becomes super clear, even more clear I would argue than OOP systems because a big point of OOP is to hide the state, and that's what makes it hard to learn and hard to reason about peoples architectures; where is this coming from and where's the state coming from and how does all of this fit together?
In a procedural style WITHOUT global state you end up having to pass everything everywhere, and that I would say is a huge self documenting feature. It just is simpler, and I really really feel that there are so many problems, architectural problems, where I don't have the state that I need and I can't get access to it; all those things that we did to ourselves in OOP; those problems simply do not exist. They just WENT AWAY when I switched, and the more I switch, to procedural style.
So I'm really finding a resurgence in my joy of programming, I love that it's all so minimal, I sort of snicker to myself "Oh look at this, it's public, it's a struct not a class... :)" All that stuff is just invigorating.
Oh by the way I went through my codebase and switched everything from class to struct, which was great because I always recall earlier days of programming when making something new there was always this sort of deliberation: "Is it a struct or is it a class?" And that's time spent, so I just went through and said "Everything's a struct!", and it just saves me time! It's all structs and functions. And if you want to do interfaces, with a C++ compiler you can still do that; a struct with methods.
The big caveat here, to be fair, is that I'm working as a solo programmer; I guess the question is, relating to larger teams: "Are there real gains to be had from encapsulation and information hiding and things that are coming from OOP?" I'm not so sure that 'procedural publicness' would be a problem for a big team, but again I can't tell because this is me running my own shop.
But I'm finding a lot of gains here, and the reasons for switching were not dogmatic or really philosophical but rather purely pragmatic, but now I'm starting to see things in hindsight that are coming out of it.
One of the things I've thought a lot about; does data in itself have implicit semantics? When we do Object Oriented stuff we think that it very much does I would argue, because here's a class and it's called Tank and it has member variables named in such ways that are obviously semantically loaded:
class Tank
{
public:
explicit Tank(const float aSpawnTime, const uint32_t aSpawnHealth);
~Tank();
private:
float mySpawnTime; //this is a timestamp
uint32_t myHealth; //this is hit points
};
Otherwise we wouldn't call those variables the things we do, and we wouldn't call the class Tank. But that's a human construct; it's not in the scope of computer science LITERALLY true that these variables ACTUALLY have that semantic meaning, it's something that we as humans do in order to express what we are trying to do in programming. But once you accept that you're "acting" like these things have semantic meaning, that doesn't automatically mean that they HAVE inherent semantic meaning.
How is that relevant? It's relevant when it comes to getting rid of constructors and destructors, which might not seem neither intuitive nor a good idea. Here's a struct from my core library:
struct blob_t
{
void* memory; //memory, not necessarily dynamically allocated, again implied semantic meaning!
uint32_t size; //size of the memory, semantically loaded in that way
};
If you had a constructor and a destructor, one way to think about the blob is that it does indeed point to dynamically allocated memory:
struct blob_t
{
explicit blob_t(const uint32_t want)
{
memory = new uint8_t[want];
assert(memory); //can fail, bad style?
if(memory)
size = want;
}
~blob_t()
{
delete[] memory;
}
void* memory; //memory, dynamically allocated, again implied semantic meaning!
uint32_t size; //size of the memory, semantically loaded in that way
};
So that's cool, because it becomes a high level concept where you get memory and it automatically cleans up.
But I want to use blob_t in the context of compression. I don't want this kind if signature:
void zlib_compress(const void* src, const uint32_t src_size, void* dst, const uint32_t dst_size);
I want:
blob_t zlib_compress(const blob_t& src, const blob_t& dst);
And (actual shipping) client usage is often like this:
uint8_t compression_buffer[COMPRESSION_BUFFER_SIZE];
const blob_t compressed_sv_packet = zlib_compress(
{ &state.sv, sizeof(state.sv) },
{ &compression_buffer, sizeof(compression_buffer) }
);
The important thing here is that there are no implied semantics of any of the memory pointed to by blobs that are passed to zlib_compress() being dynamically allocated or not, which in turn makes blob_t more reusable. This might be a trivial example (and how much code is being reused?), but you start realizing that having constructors / destructors on blob_t would only be messing with you. They would force you to use blob_t in a certain way, it implies a bunch of contracts. I guess people would say that that kind of high level construct is a good thing, but I would argue that it doesn't really make things clearer in any fundamental way, it just makes blob_t less reusable.
You CAN still use blob_t for the dynamically allocated memory use case, but then of course you'd have to "manually clean up" yourself, so you lost that stuff, but then again it becomes clear what is going in from just reading the client code, and to a certain extent it's a matter of convention and responsibility:
//clients own the returned memory... core::blob_t file_contents(const filesystem_i& impl, const char* file);
I started realizing that a lot of the things that I didn't like about OOP was things being implicit, and magical, and where's the state? Oh you don't need to care! All of the hiding. Constructors and destructors mess with me because they run implicitly. You create something, it has a constructor? It runs, so an implicit function call.
Destruction is the same thing, right? I'm not really into RAII and that sort of automatic stuff, but to be completely fair my (read) file handle abstraction has a destructor and cleans up it's own own internal implementation; I have the need for completely abstracting the file system for reading as I ship my assets in .nwf archives:
core::blob_t file_contents(const filesystem_i& impl, const char* file)
{
read_file_t handle; //cleans up internal implementation using a destructor
FS_LOG("open_binary(%s)", file);
if (!read_file_open_binary(impl, file, handle))
return{};
if (!handle.impl->total_file_size())
return{};
core::blob_t result{};
result.data = new uint8_t[handle.impl->total_file_size()];
if (result.data && read_file_bytes(result.data, handle.impl->total_file_size(), handle))
{
result.size = handle.impl->total_file_size();
return result;
}
return{};
}
Looking at that now however I'm tempted to get rid of the destructor (maybe that's asking for trouble...), but it really bothers me far less than constructors do
Getting rid of constructors really does help though, for sure. I ran into a lot of bugs when it came to the peer2 port which had to do with wiping memory. I don't like using memset() from a purely syntactical standpoint, but I really like the new C++ feature of {} initialization in order to wipe something to zero. However if you have structs of structs, and one of those structs has a constructor, using {} will not wipe the aggregate struct AT ALL due to the fact that a component of the aggregate has a constructor.
I had so many bugs when it related to uninitialized memory because there was "something down there" that had a constructor, so that was a turning point where I decided to just get rid of constructors (and destructors) in all practical cases. Sometimes I will include destructors only in debug builds in order to assert that they have been cleaned up properly by clients, but that is mainly a step in porting legacy OOP code to procedural.
This has truly been one of those liberating ideas, because all of a sudden data is just data and it has no implicit semantic meaning (ownership / cleanup of pointer members IS implied semantic meaning!). Data has semantic meaning to the extent that "This float is a timestamp.", and of course that helps you code because otherwise you'd just call your struct members int32_t a, b, c;, and that's of course hard on everyone. It DOESN'T matter for the compiler, but for you it's hard. So the point is to only force yourself to deal with as much semantic meaning as you absolutely NEED to, because being reserved makes things more flexible.
This is probably hard to fully appreciate for people who haven't gone through the journey of coming from OOP and moving to procedural, but I find it to be really important. I'm still learning, still going through this transformation away from OOP, but I think that my productivity is much higher than before.
From a philosophical point of you people might say "Oh you're doing procedural programming, that's just an arbitrary (bad?) choice over Object Oriented Programming or functional programming..." but the fact that I really feel that I couldn't have TECHNICALLY achieved peer2 without doing this switch is weighty indeed. Isolating state into one blob, getting rid of things like growing arrays, object graphs, polymorphism, all the things that I felt I couldn't have; what's interesting to me is that I really don't feel like I lost anything. I really gained a lot of insight into what programming minimally needs to be in order to function, I gained productivity and really didn't lose anything when it comes to the quality of the product (rather increased it instead).
And I'm having fun and feeling like a maverick, because of course; feeling like I'm "Going back to C..." I've gone to what a friend calls nether_snake. It's all: struct dude_t{};, a style I saw in early id Software code which I find interesting and useful. Even my constants / enumerations are tending to lowercase, not sure about #DEFINEs yet. Of course code formatting / standard has NOTHING to do with any of this, other than perhaps feeling like I'm being super minimal, and I guess that extends to the use of CAPS too...
That's really my very long winded talk about how my experience is saying; "Perhaps object oriented stuff in the scope of game development really didn't add anything critical or substantial?" For me I really feel, looking at it this way, that I've wasted a lot of time trying to be high level, trying to be abstract, trying to write "beautiful code" from a sort of human level.
Instead I'm now focusing on the fact that the job at the end of the day is that you're translating ideas, emotional concepts, graphical concepts, you're basically translating it into what the computer needs to deal with. Because that IS your job; you can try to avoid thinking about it that way and try to think about high level constructs and everything, but you still need to IMPLEMENT it; you still need to live in the byte world and you still need to live in the dynamic memory allocation world and you need to think about endianness, and word sizes, you have to think about all that stuff.
The blanket statement is: "I don't believe in OOP anymore."
The more inflammatory statement is: "I think that OOP has damaged the industry."
It has definitely made me waste a lot of time personally, resulted in some crazy architectures (that I luckily was forced to refactor to performant / shippable state), because really programming is not about the high level stuff, it's about the low level stuff, especially game development.
And these days I find I'm doing better games, better ARTIFACTS, I mean the stuff that's coming out of my programming is better stuff partly because I changed the operative procedures; I changed the CONSTRUCT; I changed from OOP to procedural, and the artifact that's coming out (this is Rich Hickey's way of putting it): I CHANGED THE CONSTRUCT and I got a BETTER ARTIFACT. So that's really the existence proof for me: "Oh, you changed the way you code and you made a better game!"
That's always going to win for me.