Network data changes

[csmith12]

Yes I too have seen an increase in warping...

As a bard I play with many mobs at once... (swarming)

When I solo I pull trains normally....

I have seen my mobs just take off running in any given direction. That I can handle, but the warping on top of me sucks.


Stun..... hit, hit, hit by all in the train = Death

Some zones are still ok, the PoD is a deathtrap for warps now for me at least.

I think they had a good idea when it comes to the patch changes, but what works in real life and in theroy is sometimes two different things.

So am i correct here?

Before patch
1. send update mob speed, direction and destination packet
2. send more packets
3. send more packets
4. send more packets

After packet
1. send update mob speed and direction packet
2. send more packets
3. send more packets
4. send mob end placement packet (warp condition)

Thanks

[Manaweaver]

I will admit things have been a bit funky since teh patch. I quad kite... I've noticed that mobs have a tendancy to not update the proper positions(making it look like they scattered) but once I finish my spell and it lands, they update back into their tight little formation. Mob warping... I think SOE has added some more code into it about when mobs have bad pathing. If a mob stays at a LOC for too long without a root on, then they get dropped on the player. I've noticed this happens whenever I circle around the mobs really tightly.

CTD's generally happen when using a custom UI... nothing fancy mind you, just any custom UI without the proper updates.

Copy the following files into your custom UI folder.

EQUI_StoryWnd.xml
EQUI_BodyTintWnd.xml
EQUI_JournalCatWnd.xml
EQUI_JournalTextWnd.xml
EQUI_JournalNPCWnd.xml
EQUI_BigBankWnd.xml

Then open the EQUI.xml from your custom UI folder with Word or notepad.

Add the following lines and or any other xml files that are not listed. Be sure to add them in with the list that has all of the other xml files.

<Include>EQUI_StoryWnd.xml</Include>
<Include>EQUI_BodyTintWnd.xml</Include>
<Include>EQUI_JournalCatWnd.xml</Include>
<Include>EQUI_JournalTextWnd.xml </Include>
<Include>EQUI_JournalNPCWnd.xml </Include>
<Include>EQUI_BigBankWnd.xml </Include>


Hopefully that helps. I've noticed my other computer(which is definately a lot more low-end machine) CTD's more often since the patch and also... it can't even ALT+Enter or it'll CTD...

[datadog]

I just logged of of EQ. It seems after the patch this AM, that the problem fryfrog described (mez'd mobs keep running, etc...) has been corrected... or at least is it much better...

I have been at the same camp the last few nights, and it was a heckuva lot better tonight than it was last night...

Also the problem with HP bars not updating for folks is better as well..

Yesterday, my cleric would show the ranger at full HP, but on the rangers screen he was at 15%.... Today they updated almost simultaneously..

AND.. packet rates were still pretty good...

[Poncho]

Manaweaver -

No need to copy those into a custom UI folder if you are using a custom UI. New patches that update the UI will put the new EQUI file in the default folder, and your custom UI will read that from the default folder. I am assuming/hoping that you dont have a custom UI that has already modified this file. If you do, you will most likely crash when new updates are made.

The safest way to run ANY custom UI is for your folder to ONLY contain the modified files. The rest will be read from the default folder if not found. This has been effective since about september or october.

I realize that this forum has nothing to do with UI skins, but I felt the need to clear that up. Regardless, Ghosting and the other problems explained in this post have NOTHING to do with your UI. Custom UI's are only for looks and have no other bearing (besides some slight memory loss if you are using huge or sloppy XML's)

Poncho

[fee]

I have hesitated to discuss these changes on public forums untill now because I really really needed to see someone else self discover the inherit problems with SoE's new network code. What Protector states is absolutly right on.

Protector wrote:
There are definite theoretical improvements in the design, ie combining small packets together and compression that make sense (these were in place before Jan. 28)

However running each opcode through a bit scrambler then feeding it through a lookup table just to get back to the old switch statement is not one of them.

A small portion of the changes to the EQ network code make a lot of sense. Here are a list of Pros and Cons of SoE's implementation as I see them.

Pros:
Combining payloads into a single packet reduces header (IP/UDP/EQ_PROTO) overhead. Total packets is directly proportional to bandwidth, duh

Compressing payloads that have repeating patterns will reduce bandwidth.

Cons:

CPU Overhead; Unfortunatly all of the Pros listed above have a consequence. SoE is essentially trading network bandwidth for Client and Server CPU. Deflating and Inflating these small (rarely large) blocks of data will over time take al chunk of CPU time. To their credit SoE is using a fast compression level (FLEVEL=1), however I wonder how many players have the spare CPU time to accomodate this, I know my CPU sits at 100% while playing. You may actually notice a pause in the client when you see the "character saved" message. When the character is saved a block of data (approx 13K) is compressed and sent from your client to the server. The zlib overhead to prepare and Deflate the data is enough to produce this small stutter.

Non self deterministic data sizes; The new network packet processing scheme requires additional calculations to determine where each payload, in a multi-opcode packet, begins and ends. This is a very minor issue, but again it requires extra CPU that does add up over time.

Stream encryption (for lack of a better term): Some people have questioned whether or not any of the recent changes were aimed at showeq. The answer is definitively yes! Without discussing the details of the mechanism in detail, I can say that this must have been designed to thwart pre-school children from parsing the packets, because thats the only audience I can think of that wouldn't "get it". So the result here is ALOT of extra processing on the packets before the client can use the data. This of course results in CPU overhead.

More implementation faults; The programmer(s) who designed this scheme forgot to take into consideration the already compressed and encrypted packets. The result here is compressed, encrypted, compressed data. Anyone that has ever tried to double zip a file knows the result, thats right it gets BIGGER! So much for some of the bandwidth savings, and hello! MORE CPU overhead.

Latency increase; No i'm not talking about network ping times. As several people have pointed out on this board, there is an increase in "ghosting" of spawns. This is the result of the servers wraping multiple updates together in combination with other types of opcodes. I can only speculate on this but logically it makes sense, atleast to me. Since the spawn updates are being delayed prior to transmission any updates that are recieved by the client are late and may not be in sync with where the server considers the spawns to be. Combined with the issue of non self deterministic data sizes, the latency increases a bit. A possible solution to this would be to increase the frequency at which the server updates the client about spawn movements and the like. Unfortunatly increasing the frequency of updates sort of negates the entire scheme. Damn double edge swords A by-product of this latency is that you may get less updates over time which would result in a decrease in bandwidth. It would be interesting to determine if this is why 70 people raid encounters do not lag you out so much.

My overall rating for this scheme would be Half-Assed. As we saw in January SoE went through 3 attempts at reducing bandwidth finally settling on the thrid attempt. The first attempt was pretty slack and lots of problems resulted. The second attempt was actually decent. But the current implementation is leaving alot to be desired.

I think it is in rather poor character to use a multi-million dollar prodction service as a test bench for a half contrived, poorly implemented "idea", is not this what the Test server is for?

Disclaimer
the above opinions are based on my personal experiences and observations while playing the game and analyzing my network data and computer cpu usage Prior to the February 4 patch. Various play styles such as raiding or sitting in small unused zones will give you drastically differing views the performance issues I have discussed.


To SoE managment/staff: This is the only time I will ever provide any type of analysis to aid your product. I'm not getting paid for this, you are. Random Tom, Dick, and Harry customer should not have to provide this pro bono(free).


References:
http://zlib.org/
http://www.gzip.org/zlib/feldspar.html
ftp://swrinde.nde.swri.edu/pub/png/d.../rfc-zlib.html

Fee
ShowEQ Developer
[email protected]

[Manaweaver]

I wasn't aware of that poncho... thank you for the advice... I'll revise my UI tonight so its that way. Should save a TON of problems =)

[wfj5444]

Very good post Fee! Nice to get some inside info on the way things are happening.


Thanks greatly!

[cbreaker]

Unfortunately, this probably means that in the future they will, again, completely revamp the system.

Some new ace programmer will sit down with the net code, and say "WTF is this crap?" and code out something that we hope will be good. Unfortunately this would break seq again heh. I mean, sure, companies keep crap code in their software all the time, but for something like an online-only game, it doesn't seem too likely that they will keep "Half-Assed" network code in there forever.

Chunking the data together is a good idea; and in a raid situation it helps quite a bit. When you have large amounts of data coming in, this is where it performs very well.

The only way I can see that they could keep chunking data and at the same time keep movement updates relatively smooth, is to have a seperate connection for the stuff that you want to be updated asap, which wouldn't be chunked so much. But I ain't no expert. I'm sure there's an easier way.. one of them being to stop with all this fancy-dancy stuff to thwart packet sniffing.

I guess it's the same old arguement about copy-protection. No matter how hard you try to keep a peice of software protected, someone will crack it. So does that mean you stop trying to protect your software? IMO yes, stop it's a waste of money. But for the company brass, they CAN'T stop because they need to keep collecting paychecks..

[S_B_R]

Thanks for the info Fee. I just wish someone from Sony would actually take what you've said and do something with it....

[cattj]

An overall evaluation of the network protocol ( ie how often opcodes are sent vs. the size of the opcode) added to the priority of the opcode being sent would give you a good idea of how beneficial it is to have this packet wait for others in order to send itself in a chunk. Using this data the queueing mechanism for setting up these chunks can take opcode priority into account.

Warping is going to occur in any case even if they sent every location update packet immediately. The client is attempting to simulate something that is happening in real time on the server. There is no network protocol in existance that can achieve this over a real network.

Now I haven't looked at the network protocol for EQ in ages so I don't claim to be an expert on the exact implementation in question here. Are they chunking together multiple loc updates for a single mob? Or is the server doing the logic where by an old loc is really useless therefore not included?

In the easiest multipacket protocol you would essentially have a time and/or a size threshold. If this packet reaches x bytes then send it and/or if this packet hasn't been sent in y seconds send it. Both of these are however, quite stupid when it comes to the actual payload of the multipackets. Knowing the protocol allows you to weed out redundancy and set priorities.

Aside from the disconcerting feeling of mobs jumping around the server is still running its checks the same as it always has and so gameplay (aside from aesthetics) should not be changed. I think SOE was trying to trade off lower bandwidth for smooth gameplay and it seems from most reaction that they moved the bar a bit too close to the lower bandwidth.

Priority queuing on an opcode basis would be help(do they do this already?) where by they allow themselves to alter the x and y in the above chunking example to better reflect the urgency of the payload.

[Amadeus]

I hate to put such a simple viewpoint in a great, technical discussion (gratz Fee!)...but, am I the only one that doesn't notice any great difference other than negative changes?

Things worked just great for me before the patch, even with huge raids and in the bazaar. It's more of the same now, except with a lot more ghosting and weird screen flickering.

I just wish they'd fix bugs in the game, or add content -- adding more bugs to the game really doesn't make any sense.

[bonkersbobcat]

For me the problem hasn't been the performance of the network protocol, it has been the performance of the game itself. I have DSL and and when playing EQ I never max out my network feed. I always, however, max out my CPU when playing EQ. IMO it would be effort better spent by SOE to improve the performance of the base game rather then screw with the network.

[darkgrue]

IMO it would be effort better spent by SOE to improve the performance of the base game rather then screw with the network.

/agree

The lastest patch not only messed with the network protocol, it also made some changes to the DirectX interface to the game, which manifested itself with The Return of the Stupid Phatom Mouse Cursor, and tons more desktop bleed-through. How that sort of garbage could get through even the most rudimentary test mystifies me, unless it's unique to NVIDIA cards, and Verant doesn't test on probably the single most popular gaming video card...

The rendering engine alone needs help. All you need to do is do a quick (err, well maybe not-so-quick) 360 in the Bazaar to see that it's thinking really hard about a lot of stuff that you can't see. Which isn't trivial optimization, but it certainly would be an effective improvement to the game's performance! Luclin was supposed to improve the renderer with the inclusion of the Umbra engine so that the game could avoid rendering hidden surfaces, but whatever method that engine uses doesn't seem to be terribly effective.

[S_B_R]

Originally posted by Amadeus
I hate to put such a simple viewpoint in a great, technical discussion (gratz Fee!)...but, am I the only one that doesn't notice any great difference other than negative changes?

Things worked just great for me before the patch, even with huge raids and in the bazaar. It's more of the same now, except with a lot more ghosting and weird screen flickering.

I just wish they'd fix bugs in the game, or add content -- adding more bugs to the game really doesn't make any sense.

I've only seen negative results as well.

Originally posted by bonkersbobcat
For me the problem hasn't been the performance of the network protocol, it has been the performance of the game itself. I have DSL and and when playing EQ I never max out my network feed. I always, however, max out my CPU when playing EQ. IMO it would be effort better spent by SOE to improve the performance of the base game rather then screw with the network.

I also couldn't agree more!

[uncleubb]

I find it fascinating that a person who is a "mere" customer obviously has a better understanding of the product than the product maker.

If my development team was put to shame like that by a consumer, I would be extremely embarrassed. It would be time for firings and hirings... in that order.

I know fee is obviously very smart and knowledgeable, but he can't possibly be the only person with this kind of networking savvy... (in fact it appears to me that numerous such people regularly participate in this forum).

If SOE did this to spite SEQ, they really ought to consider serving the VAST majority of their customers before targetting a very small minority. Macy's doesn't pat down every shopper just to make sure they didn't steal anything...

It's a simple matter of quality and service.

And just in case SOE has forgotten, or thinks these terms are some unknown foreign speakease (and because I love to post definitions):

quality

adj 1: of superior grade; "choice wines"; "fine wines" "prime beef"; "prize carnations"; "quality paper"; "select peaches" [syn: choice, fine, prime(a), prize, select]

service

n 1: work done by one person or group that benefits another; . . . 13: the act of mating by male animals [ok, not that type of service please!];

-uu