A Saturday Morning Update – The 90 Day Plan – Review, Update, and Tease!

Folks,

Greetings all! Here’s something we don’t do all that often, a Saturday morning update. Yesterday got a little crazy time-wise at the end of the day, so instead of writing this up late yesterday, I decided to get it done this morning. The focus of this update will be on the first 90-day plan, next week’s testing plan, and a small peak into what we will be doing over the next 90 days. There’s a lot to cover in this mini-update so let’s get this started.

First, as you Backers know Covid has affected us, as it has the world as the studio is on WFH mode. While the team is working hard and delivering at a very high level, it is still a stressful and difficult time for all of us at CSE and elsewhere. However, even under those circumstances we have a lot of great news on what we accomplished on the 90-day plan and we have high hopes for the next one as well. We’ve been running “All Access” testing with our latest and greatest builds almost every day during the week, giving more of our Backers access to the game than ever before. As a result of that we’ve been getting a ton of feedback and iterating on it quite quickly. One of the areas that has been of main concern is the performance of the engine’s rendering capabilities when we let the artists go nuts and drop lots and lots of things into the build. Because of the feedback from Backers coming in with powerful, but 2-generation old cards (GTX9xxs) I asked engineering to do a deep dive into the renderer to see why our performance was acceptable but not optimal. We started down that path on Friday, found and dealt with some issues but not in time to get a build out for the weekend. We will continue this dive next week. The good news is, of course, that even in its less optimal state, the RvR scenario was still happily dealing with 14M blocks, players, etc. effectively, if not at an FPS that we want to see in the finished game. The other good news is that this has nothing to do with our ability to network these large-scale battles, even in areas that aren’t as full of trees, grass, props, etc. And as I said, we already found and fixed one problem and I expect more fixes incoming over the next two weeks. Please expect another full week of testing this coming week which I’ll talk about a little more below!

Second, back in February I laid out a 90-day plan for Camelot Unchained that showed our commitment to not only making a great game but also doing it in a more timely manner that people expected from us. And now, 90 days later I’d like to go over with you, in excruciating detail, how we did with that plan. Of course, what follows are just the items from the 90-day plan. Other CU-based things were done as well that weren’t part of the plan but came up during the work we were doing on the plan. Because this plan is already way-too-long, I didn’t ask the team to cover them here.

Camelot Unchained 2020 90-Day Plan #1

GENERAL

  • Visual update to Cherry Keep
    • With updates to rendering we made 2.0 material updates to all characters and gear. This also included terrain material as well as general fixes, lighting improvements as well as upgraded CUBE materials. Visual upgrades were implemented on cliffs and rocks taking advantage of newly gained tech.
  • Visual Improvements to CU using new Engine Technology
    • Recent additions of tools were world aligned textures and vert painted textures. Our engineers have been working closely with art and gameplay to find efficient opportunities to upgrade the way we work and the quality of content so that we can keep the development train moving forward.
  • New CU Scenario
    • RvR Battleground Scenario
      • The new RvR scenario has been going through testing the past two weeks. The general game loop of keeps, towers, and keep lords have been implemented and the realm ownership system is working to keep the score of the scenario. A suite of new systems have been implemented to enhance this gameplay experience. (Soul essence shards, spawn protection bubble, and in-game scenario map with objective tracking.) We’d also like to thank the Builders Brigade on working closely with us and providing some amazing buildings across the scenario. Shipping what we have today could not have been possible without them.
      • New Siege Engines: We made a great deal of updates and fixes to the siege engines. Interaction points have been improved and we have continued to iterate on their damage against buildings. They were also updated to show visually unique projectile models and improved projectile VFX were made.
    • Scripted events using Overmind
      • The new RvR battleground zone is controlled using the Overmind which was a major goal of the development of this scenario. This includes test scripting so that with a press of a button we can admin spawn NPCs and siege engines. We have so many ideas and features we’d like to continue down this route and we hope you are as excited as we are to see them in the coming future. Additionally, late on Friday we landed an update to Overmind to allow it access to all parts of Camelot Unchained and not just the RvR-scenario as well as to tie it into specific elements of the game.
  • Updates to CU Characters
    • Updates to abilities
      • Channeling and an update to trigger volumes enhanced some of the older abilities as well as paved the way for the set of devout abilities. We are hoping to utilize this a lot more in the future.
    • Updates to art
      • Sweeping material fixes and updates across the board for characters. We also brushed up on a suite of capes and skirts. Not to mention the process and material changes for environment art.
  • Warbands
    • First iteration of this feature has been implemented! Now you can band together with your comrades to battle it out in the new RvR Scenario. We’ve also worked on a system such that certain abilities now only apply to your warband, with more planned for the future.
  • UI Updates
    • We made some polishes to the character creation UI. We implemented an update to the ability bar system and though we have been working through bugs on that front we hope that the customization opens up new ways for players to interact with their abilities.
  • New class trio – The Devout
    • The devout trio is out and under grueling testing. With the help of backers and testers our development team has been able to work through a great deal of bugs and balance iterations with these classes. We learned a lot internally about our process of building new characters in CU and feel like we have some improvements in the pipeline to bring you in the future!
  • Planning for next class with consideration of bringing in of an “Extender” class, Spirit Mages
    • Ben has put a serious amount of time laying out the design for this next class trio. Our hope with cleaning up the support trio and shipping our first pass of the devouts and iterating on them is that we pave a better process to designing, developing and shipping characters in CU.
  • Review of crafting system and next steps
    • MJ completed a lot of background work for crafting but due to engineering time constraints (he want(ed) the team to focus on RvR/Devout) there wasn’t enough time in the schedule to implement this aspect of the 90-day plan. Crafters and crafting will be a larger part of the upcoming plan.
  • Live Service Stability and Scaling
    • Lots of support work done here in terms of continued migration from AWS to GCP for all of Camelot Unchained. More on this below.
  • Additional Lore
    • As part of the crafter update next month MJ will spend some additional time on lore and the upcoming Giant race.

ENGINE

  • Client Memory and Performance Improvements
    • We have made a number of improvements to the engine but still have a ways to go. Late last week Andrew landed an update to the engine that allowed it to better utilize all the PC cores/threads on the rendering side. These improvements can be seen in the Home Islands where FPS has improved.
  • Path for older video cards
    • While some work was done, more is needed as well as work for those folks with AMD cards. This will be a focus on the next 90 day plan.
  • Linux Focused server performance improvements
    • Got Linux servers running at full speed on target configurations
  • Better graphics configuration
    • We got some things done in terms of work on the editor, this will get more attention in the next 90 days.
  • Improved 3D Pathfinding
    • Navmesh generation and pathfinding
      • Continuous improvement to the types of geometric cases our system will handle.
      • Able to generate paths up some types of stairs and ramps
      • Added partial-path determination, that can return an intermediate solution that’s en route to the full solution w/o committing the resources require to compute the full solution (allows NPCs to start moving towards a destination without having the complete path yet)
      • We are still working towards a holistic, consistently working solution for navmesh generation in general, which at this point means testing different use cases and identifying bugs that come up. The work remaining will continue to focus in this area, with numerous examples documented that serve as working test cases to be solved.
    • Path following
      • NPC behaviors incorporate ‘partial path’ capability to find nearest positions to their destinations, in cases where the destination is inside world geometry
      • Added NPC stuck detection, and ‘get unstuck’ logic that improves the ability of NPCs to get themselves out of situations in which they cannot move
      • Improved logic to time-slice NPC path requests such that few paths are requested while preserving the same observable level of NPC navigation success
  • Overmind Upgrades
    • We’ve developed the ability to run Overmind scripts without the need of a scenario so that there is a future where queuing into a scenario won’t be necessary. We’ve also added a suite of new actions and triggers and swept through a list of bug fixes.
  • Ragdoll Improvements
    • Fixed issue with corpses sliding too far
    • Fixed issue with skeleton bones that do not participate in ragdoll causing incorrect positions of ragdoll-participating bones
    • Fixed ragdolled corpses laying perfectly flat at all times
    • Improved performance over original implementation
  • Animation Improvements
    • Added 2-bone skinning, which provides higher fidelity skinning w/o suffering the performance cost of full 4-bone skinning at mid-range LODs
    • Fixed jumping then landing while running interrupting the animation cycle, such that you can now jump and land with seamless flow into the run
  • Player to Player Shoving based on mass and speed
    • Earlier in the milestone we had begun investigating implementation of this tech. Our investigations took longer than we estimated them to be and the result and outcome was that it required a lot more development time than we could afford in light of shipping the RvR scenario. We’ve put this in the backburner for now but the good news is we have a lead on how to tackle this in the future when we can come back to it.

OPERATIONS

  • Server Orchestration and Automation Tools
    • There’s been a big focus on improving and extending our Locust testing framework. We’ve built numerous Locust tests for a bunch of our systems so we can automate testing against them to test scalability, stability, and regressions. This has helped us find and fix several issues related to WebAPI, Patcher, Login, Groups, Warbands, and Matchmaking.
    • We’ve improved our cloud proxy integration to automate spinning up of proxies for dynamic runtime scalability.
    • We built and deployed our service discovery solution as a major step towards general improvements in automation, scalability, and self healing of our server stacks.
    • We did scale testing of our server allocator allowing us to find and fix bugs and build confidence in it as a live-ready product.
    • We implemented and deployed our Remote Config system which allows us to trivially reconfigure a wide range of settings on a wide range of servers at run time. This will eventually reduce the number of configuration files we manage; significantly reducing the surface of configuration errors and automating the process of deploying them.
  • Build CI/CD Pipeline improvements
    • We whiffed on this one, just ran out of time.
  • Beginning of Live Operations Tooling
    • We’ve integrated a hodge podge of one-off tools including the AdminTool and GlanceView into a single cohesive interface. The new admin tool has been improved, extended, and maintained to be a first class tool rather than something sitting off in a corner.
    • Admin tool has been deployed to an instance group backing a load balancer for high availability to developers. This is still a WIP.

I know, that’s a lot of information to digest but I thought it was important to share *all* the information with you folks, rather than something as simple and uninformative as either percentages or a Pass/Fail system or ways to disguise our misses and enhance marginal success. As Backers, you deserve more than that from any Kickstarted game, and we have given it to you. Now, what does all that mean? It means that we accomplished the vast majority of our goals with only a couple of exceptions. Depending on how you want to rate/score it, that’s a pretty good result for any team, let alone any team working in these more difficult times. It was one of our most effective periods as a studio and we hope to continue with the same trajectory in the next 90 day plan. Back in February I promised you that we were going to do this and well, we did. I hope that you enjoy seeing these kinds of updates from us, because more are coming.

Now, let’s have a brief chat about the next 90 day plan. While the team is still filling out the details, there are a number of items that are extremely important to me that will be a part of the plan. Not the whole plan by far, but just some key things I’d like to give you ahead of time.

1) Performance – This will be a major goal for the team. As you saw over the last 2 months, CU went through a beautification pass. As part of that continuing effort, we must make sure that even with all the new stuff we have added, that the game’s performance gets back to where it was prior to that pass. We also have to look at performance on AMD cards as well, since they need some love.

2) Crafting – I want us to begin work on the crafter class to coincide with the work I was doing on crafting systems. We will also look at the crafting UI, the Vox, etc. I hope that at the end of the 90 days we have a fully playable crafting class.

3) Scouts – Along with crafters, work will continue on the Scouts.

4) New Siege Scenario – I want us to begin to spec out what a true siege can play like in a smaller scenario than the current one, before the implementation of the first stage of the Dragon’s Web. The scenario will be designed to be short, self-contained and emphasize siege weapons, building destruction, and crafter support (if possible). We’ve got some great castles from the Builders’ Brigade, time to spend more time blowing them up.

5) Bug Eradication – We started focusing even more on this last week but I want this to continue through the next 90 days as well.

6) Class Balance – Ben has been working on this throughout the last plan, and it will continue in this plan.

7) New Races – The Giants will be the first of the new race trios to be added to the game. Because of their size, they will present some challenges that need to be addressed sooner, rather than later. We know we can easily do bigger characters, but I want to see how far we can go before we start to really mess things up. I’ll also decide what the next race trio we will be working on is.

8) 24×3 servers – And last, but certainly not least, one of our main goals for the 90 days is to get CU in a state where we can run it over a full weekend with only minimal oversight by the team.

That’s enough for now I think. Expect a full 90 day plan in the coming weeks which will detail all the things we will be focused on as a team. Just as I made a commitment to our Backers back in February that CU would continue to be the focus of the studio, I’m making the same one now. Barring some extreme circumstances (given what’s going on these days, anything is possible, I just hope that the Stay Puft Marshmallow Man doesn’t make an appearance) we’ll continue to push hard and fast on CU as we just have done this past 90 days.

In terms of our testing plan for the coming week(s), expect all access testing to continue during the week especially as we work on performance improvements. As soon as we get those, we’ll open this up for weekend all access testing. We have a lot of things in flight so it should be a busy and fun few weeks.

And with that, I want to thank you for your support and patience especially during these difficult times. We at CSE (team and investors) are committed to delivering on Camelot Unchained as quickly as we can while delivering on the kind of game that our Backers want to play and play a lot. It has been a long and difficult road but as this past 90 days has shown, we can deliver on what we said we would even while living in CoronaWorld. From us, no excuses, no whining about the difficulty, just continued work and effort and we’ll tell you where we succeeded and where we didn’t. Expect more great things this coming quarter, I know I am.

As I’ve been saying for the last few months, please stay safe, uninfected, and be kind, if possible to others.

-Mark