Folks,

Well, it was a very interesting 72 hours of weekend testing for our Alpha, IT, and Beta 1 Backers. The bonus edition of our Friday Night Fights got off to a great start! For seven hours, the servers were rock solid. Backers were building walls, hurling elemental balls across the land, and everybody had a great time. We were so confident that we found and fixed the heap corruption bug that when almost everybody left for the day, they did so with a smile. Unfortunately, at a bit past seven, the server hung. So, for a bit we thought we had been wrong about fixing the bug, and we were sad ducklings. With a mostly empty office, we set about figuring out what went wrong. We quickly found the problem, and, much to our chagrin, we couldn’t believe what caused the problem! Pop quiz, hot shots, was it:

a)    Heap corruption
b)    Thread contention resulting in server freezing
c)    Crashing due to any other bug
d)    Running out of hard drive space on our cloud-based servers
e)    Internet dying due to Kim Kardashian prepping another bottle of champagne
f)    Verizon, well, being Verizon
g)    None of the above

And the answer is – (d).

Yes, our good and noble Backers, we ran out of hard drive space, on our AWS servers, in 2015! Can you believe that? Of all the things we didn’t expect to happen, well, it did. We had been running Bot tests for the last few days, and well, our Bots were very verbose and productive, and one of our logs filled up. Yep, that’s all it was. If this was a headline for Variety or Hollywood Reporter, it would read “Busy Body Bots Break Build!” 🙂

For our Backers that were around that evening, you know that we got things back up and running nicely before midnight, so the first 24 hours weren’t a total loss. We then extended the FNF for another 48 hours, as a way of saying “thank you” to our Alpha, Beta 1 and IT Backers. We also hoped to see if we really had solved the heap corruption bug that had been eluding us thus far.

Now for the good news. The servers ran flawlessly for the remainder of the test period (almost 60 hours). No crashes, no hangs, no big slowdowns, etc. Rock solid performance all weekend. No server or mini-server restarts, either. Just our servers doing their thing, and even with the debuggers on, performance was great. So, based on everything we’ve seen during the week and then on the weekend, we are confident that we have indeed slain the nasty heap corruption bug! Yay team, and thanks to all our testers!

It’s at moments like these that our Backers should be proud of themselves, and of CSE as well. Having Alpha Servers running all weekend without fail is a pretty considerable feat for a team that is building almost everything from scratch. Yeah, it would have been nice to not have run out of hard drive space (still asking myself, how does that happen on a cloud-based instance, in 2015?) but other than that, it was a great weekend.

As I’ve said before, and I’m not sorry to repeat myself, our client/server tech is more reliable, for this stage of development, than any other game I have ever worked on. This is by no means meant to discredit the work I or other teams have done before, because frankly, the Dark Age of Camelot team was great and true, and the game was also remarkably solid, even during Beta. However, the difference is that this time we have built our own rendering engine, using a physics system that’s usually client-side on a server, and have client/networking code that can already handle larger-scale battles without it turning into a slideshow (and the battles it can handle are only getting larger!).  And also unlike the past efforts, this team didn’t have years of code to use as our base; so we truly had to build almost everything from scratch! And thus, doing what we have done is a pretty nifty accomplishment, if I do say so myself. Again, though we are still in Alpha, Andrew and the team have already worked wonders.

So, just a quick not-a-weekend update from all of us to all of you. As the days to Dragon Con dwindle, things are moving along quite nicely here. We hope to see some of you there, this week! And if not, well, our Alpha, Beta 1, and IT folks will see lots of new stuff in the coming weeks, including the class reveal presentation Part 1 (it’s a multi-part reveal, we do have a lot of classes), and a new terrain map. Oh wait, was the bit about the “lot of classes” a foreshadowing?

And for today’s bonus piece of concept art, here’s another piece where Sandra breaks down the various pieces of armor that players will be able to craft and wear on their characters. Our crafters will be very busy, with lots of different things to make.

I hope you’ve had/are having a happy Monday. It’s been great, here!

-Mark

P.S. Here’s another screenshot from the new area that we are working on for Dragon Con and beyond. Step by step, inch by inch, the tech that we need to build the world comes together.