COMMUNITY - FORUMS - GENERAL DISCUSSION
Major development changes being considered for 2020

Note: Information provided comes from public sources, and is not under NDA to the best of my knowledge.

When the COE Kickstarter launched SpatialOS was a cornerstone of the design which was going to help bring the game quickly to market.

In 2018 SBS decided to not use use SpatialOS for several reasons, instead replacing it with a number of in house / package solutions.

These included using RabbitMQ, Docker Swarm, NodeJS, PostgreSQL and VoxElyria.

Heard over on the Grapevine recently, Caspian shared some information which indicates another major shift in development direction & tools is currently under investigation.

Team is investigating using C++, C# or Rust instead of NodeJS as its process overhead and large memory footprint make it impractical for creating large scale worlds like COE.

Docker (Swarm) will likely be replaced with Kubernetes. (it's what all the cool kids are doing these days, even where I work)

They are also considering other programming languages for the service mesh.

While they currently are using RabbitMQ for messaging protocol, they are experimenting with alternatives atm.


You must all be quick to listen, slow to speak and slow to get angry. JAMES 1:19 NLT

...
2/16/2020 8:04:14 PM #136

Posted By Kyleran at 09:21 AM - Sun Feb 16 2020

I think you are down playing it too much, it was pretty clear from his comments he's doing research because he feels there's a need do so as some of the current solutions aren't going to work.

We know you are attempting, again, to ramp things up into hysteria since you see a new opportunity after failing the last time.

Kudos for tenacity, but accept the fail and move on.

2/16/2020 8:25:13 PM #137

Posted By Tulloch at 12:29 PM - Sun Feb 16 2020

It’s not contradictory, it is unofficial therefore it makes no sense to give new information there other than pings for official announcements with links for where to get more information. The issue is they need to be posting things here. People shouldn’t be forced to go to a separate platform when the company has their own website and forums. Not to mention the fact that some people are unable to get to that platform at this point. At the end of the day we as a community should not roll over and accept the downright dismal communication we have been consistently getting

Its not information. A choice hasnt been made. It was casual discussion. The problem isnt him doing that, the problem is folks not being able to control themselves in reaction to that.

Right here in this post you talk about dismal communication while at the same time arguing for why he shouldnt be sharing what he is thinking about doing BEFORE EVEN DECIDING ON WHETHER OR NOT TO DO IT. Yes, it is a contradiction.


2/16/2020 9:01:53 PM #138

Us asking casual, well-intended questions (as these certainly were) in Discord should be fine. Caspian answering them without a sworn deposition, even in Discord, should also be fine.

But it's really unhelpful for any community members, or prospective community members, or even tourists, to see how close to the line of yelling "FIRE-DRAMA!" in a dev-theater they can get before being banned or modded.

There was a totally appropriate conversation about these technical programming issues to be had here -- which some thankfully engaged in, despite the noise -- but it was overshadowed by the implicit and explicit drama.

People who can't/won't even frequent Discord should not be complaining about a lack of context, like their absence creates a super-obligation on everyone else's part to compensate.

Let's just keep the drama and brush-fire match-lighting out of the thread titles and conversations.


2/17/2020 2:12:19 AM #139

When the updates on their forums and site are near dead silence, that is when its not ok to give loose updates carelessly with essentially no context. because those are the times when that is all we have to go on


2/17/2020 2:28:38 AM #140

Posted By Tulloch at 7:12 PM - Sun Feb 16 2020

When the updates on their forums and site are near dead silence, that is when its not ok to give loose updates carelessly with essentially no context. because those are the times when that is all we have to go on

I certainly understand the thought behind this. Once again though, it wasnt an update. It was pure conversation and it was started by folks who came across the job opening descriptions found on LinkedIn. I think its most definitely fair to question them looking into new programs to build with at this point. I dont think its fair to try and tell Caspian to not come around his own discord and to not have conversations about what he is thinking of doing. We now know about these potential moves because of that AND we are talking about it on the forum.


2/17/2020 5:31:55 AM #141

Posted By 1mmaculateDeception at 9:28 PM - Sun Feb 16 2020

Posted By Tulloch at 7:12 PM - Sun Feb 16 2020

When the updates on their forums and site are near dead silence, that is when its not ok to give loose updates carelessly with essentially no context. because those are the times when that is all we have to go on

I certainly understand the thought behind this. Once again though, it wasnt an update. It was pure conversation and it was started by folks who came across the job opening descriptions found on LinkedIn. I think its most definitely fair to question them looking into new programs to build with at this point. I dont think its fair to try and tell Caspian to not come around his own discord and to not have conversations about what he is thinking of doing. We now know about these potential moves because of that AND we are talking about it on the forum.

Once the marketing person gets on board this will not happen. It is literally the worst way to communicate because you lose control of the narrative. Instead of SbS putting context to the issue we get a forum post titled Major development changes being considered for 2020. And this is the first that 99% of people will see of the issue. Whether you think that's a good thing or not is irrelevant because I can guarantee that a marketing person, even one fresh out of college would lose their shit if that is how potentially major news was disseminated.


2/17/2020 8:53:09 AM #142

Posted By Drudge at 06:31 AM - Mon Feb 17 2020

(...)Instead of SbS putting context to the issue we get a forum post titled Major development changes being considered for 2020. And this is the first that 99% of people will see of the issue(...)

That's why SBS has decided on the 'rule' not to quote them on the forum. It's a tricky subject and I'd guess one of the things where they can't win and make everybody content.

One of the things the community could do about this is, whenever something similar comes up, contacting some of the 'content creators' (e.g. TTC, BBC,...) with access to the 10k Q&A, trying to get an own 'subject matter expert' to phrase questions, convincing SBS to be allowed to discuss answers publicly on their show, followed by a forum posting with a short summary and a link to the video. That solves multiple issues, first of all, it grants content creators subjects to discuss (and directs audience to their show), the 'general audience' gets a more detailed explanation and interpretation by tech savvy community members and the additional overhead (work) required by SBS is negligible (they'd have to do the Q&A regardless, I think, I'm not part of the 10k club and have no real information on it).


Sage willing to help with Purity (if you spot me on Discord and have some Plague on your account that could be nullified with a trade, drop me a message on Discord)

2/17/2020 3:09:28 PM #143

Can’t say “it’s not an update” when it’s the only information we are getting


2/17/2020 3:15:21 PM #144

Posted By Tulloch at 02:09 AM - Tue Feb 18 2020

Can’t say “it’s not an update”

Yes. Yes they can.

Threads gone to shit now, but the earlier posts were pretty interesting. I wasn't ready to jump to any conclusions based on Discord banter but what the OP appeared to imply was that they might have to scrap a chunk of work as they make changes. I'm glad that's not the case and people were here to help clear that up.


"If we wait until we are ready, we'll be waiting for the rest of our lives..." code: CD83B4

2/17/2020 4:09:41 PM #145

Posted By Pendulum at 10:15 AM - Mon Feb 17 2020

Posted By Tulloch at 02:09 AM - Tue Feb 18 2020

Can’t say “it’s not an update”

Yes. Yes they can.

Threads gone to shit now, but the earlier posts were pretty interesting. I wasn't ready to jump to any conclusions based on Discord banter but what the OP appeared to imply was that they might have to scrap a chunk of work as they make changes. I'm glad that's not the case and people were here to help clear that up.

Without something from the studio it isn't cleared up, we can't even be sure what exactly Caspian meant with how vague the statement was.


2/17/2020 7:10:59 PM #146

Posted By Logain at 3:40 PM - Thu Feb 13 2020

Posted By arki at 8:31 PM - Wed Feb 12 2020

(...)Moving from swarm to kubernetes makes sense - as that is becoming a defacto standard with docker deployments (...) there really shouldn't be any issues with doing this transition(...)

Since you seem to have first hand experience, I wonder if you have ever benchmarked (or know of an up-to-date benchmark) the startup times for Swarm vs Kubernetes? I only know horribly outdated results, most unfortunately, and would be curious if Kubernets managed to significantly improve in this aspect (or the new Swarm mode became worse then the original standalone)? I'm not sure if a 2 second startup time would work for load balancing areas/instances in a MMO?

Interesting point there. I have never benchmarked those no. Our min/max load is pretty stable across time for a given customer so we havent really dived deep into autoscaling. This will probably change as we get more experience with kubernetes.

I would think that startup time shouldn't really need to be an issue in theory unless you have insane ramp-up time. I mean it should be possible to allways warm up X number of instances in advance and just enable them as load increases. Don't know if this is something kubernetes actually supports though.

Ofc an MMO might have insane peeks i guess which could in theory make container startup time an issue.


2/17/2020 7:14:11 PM #147

Posted By Logain at 02:53 AM - Mon Feb 17 2020

That's why SBS has decided on the 'rule' not to quote them on the forum. It's a tricky subject and I'd guess one of the things where they can't win and make everybody content.

I don't understand the rule personally. I think it does more harm than good because people can't share what was actually said, only their interpretation of what was said. And that's very dangerous.

For the purposes of discussion, it's my opinion that we should be able to share anything said in a public Discord channel. It's public, after all.

Community created topics in the forums are no more official than community created comments in Discord. The challenge with Discord however is that it's a disorderly — and often very noisy — place where comments are interspersed with random other thoughts from random other people so conversations that involve many people quickly descend into chaos. Imagine a room where everyone is talking at the same time. That's discord. That's why it can never be as good as the forums for communicating important things coherently. And that's why we need to be able to talk about things here in a more discussion friendly format.

That's my 2 bits.


2/18/2020 3:13:54 AM #148

Posted By Tulloch at 03:09 AM - Tue Feb 18 2020

Without something from the studio it isn't cleared up

"It"?

Myself and a few others in this thread have had a small handful of things clarified and cleared up. Please don't try and speak for me. I meant what I said.


"If we wait until we are ready, we'll be waiting for the rest of our lives..." code: CD83B4

2/18/2020 1:32:48 PM #149

Posted By arki at 8:10 PM - Mon Feb 17 2020

(...)unless you have insane ramp-up time(...)

I'd guess that this is very use case specific. For a web application, 2 seconds is annoying, but likely nothing to warrant a redesign. For an MMO, 2 seconds of latency is going to massively annoy players though. And more than 2 seconds (some of the instances seem to have taken 15+ seconds) is likely no longer acceptable.

Posted By arki at 8:10 PM - Mon Feb 17 2020

(...)it should be possible to allways warm up X number of instances in advance and just enable them as load increases. Don't know if this is something kubernetes actually supports though(...)

Certainly. Even if Kubernetes doesn't have that built in, you can emulate a buffer by yourself. That said, if you have 200k pods per 'server/world' and require a 10% buffer, that'd mean running an extra 20k pods. That's already a considerable extra fee there.

Posted By arki at 8:10 PM - Mon Feb 17 2020

(...)MMO might have insane peeks(...)

The problem is, we don't know how they are going to handle OPC/NPC containerization. If only the 'gameplay' servers are going up and down semi frequently, this is certainly a non issue. But there's roughly 500k NPCs (according to D&SS) on the starting continent, there's a good amount of 'animals' and there's an unknown amount of NPCs/animals on the other continents. There's likely some very intense (day/night) fluctuation there and we don't know if they plan to reuse containers by simply re-initializing them (a possible security vulnerability, as they intended to use containers as security measure), or even if they are going to run them persistent for each entity (you'd just stop and run process in a usecase scenario like lucet in order to save resources). All in all, it's just something that might be good to keep an eye on, especially considering that Nodes memory consumption seems to have come as a surprise for them (which quiet frankly, no offense intended, it should not).

Posted By Hieronymus at 8:14 PM - Mon Feb 17 2020

(...)The challenge with Discord however is that it's a disorderly — and often very noisy — place where comments are interspersed with random other thoughts from random other people so conversations that involve many people quickly descend into chaos(...)

I wholeheartedly agree with you that discussions on Discord are a mess and the forum is far superior for that. My only problem is, I love receiving the rare information and insight and I can understand if SBS doesn't want them to be taken to an audience that might not be willing to take that sort of information as it is intended. I somewhat see it as a price to pay for the option to have that information. Does that make sense?


Sage willing to help with Purity (if you spot me on Discord and have some Plague on your account that could be nullified with a trade, drop me a message on Discord)

2/19/2020 5:35:34 PM #150

Isn't this related to the planned scale (number of ingame assets) of COE? Caspian has been very transparent both about the planned scale (which is unprecedented in size) and about his inability to achieve that scale thus far. I would be more concerned if he rolled back the goal or kept beating his head on a technical wall than I am with his presenting a plan to try a different technical approach.


Count of Frostale, in the Duchy of Fioralba, in the Kingdom of Ashland, by the Grace of Haven. The above opinions are mine alone and do not reflect those of my Kingdom or Duchy.

https://chroniclesofelyria.com/forum/topic/17117/naw-the-duchy-of-fioralba https://chroniclesofelyria.com/forum/topic/14124/naw-kingdom-of-ashland https://chroniclesofelyria.com/forum/topic/30605/of-contracts-and-commerce-a-tldnr-post https://chroniclesofelyria.com/forum/topic/31835/on-taxes-rents-and-ancestral-lands

...