ShowEQ 4.3.18 unknown spawn discussions

[Alfred]

First and foremost; Great work Zaphod and I hope you conitinue your work.

Second, I have noticed in HoH and at least a few other zones, that I end up with unknown spawns (as has happened in a previous release until Zaphod nailed it) with the latest release. This happens at some point after I have been in the zone for a few minutes. Once it starts, everything new becomes an unknown.

I am using:

ShowEQ 4.3.18, Built from 'main.cpp' on Nov 11 2003 at 07:57:33
CVS: $Id: main.cpp,v 1.33 2003/10/25 00:20:59 dohpaz Exp $
Using GCC version: 3.2.3
Using glibc version: 2.3
Using Qt version: 3.1.2
Using headers from linux version: 2.4.19
Using EQItemDB: @(#) $Id: itemdb.cpp,v 1.10 2003/10/25 00:20:59 dohpaz Exp $
Using GDBM: This is GDBM version 1.8.0, as of May 19, 1999.
Running on Linux release 2.4.20-gentoo-r7 for processor i686


I would be happy to provide a capture log if you have the time and inclination to track this down. I hope others (if there are any) will post if they are seeing the same thing. If you do want a capture log, give me a quick description of how you want it done. Thanks.

Regards,

Alfred

[Alfred]

So I assume that I am the only person to see unknowns?

I admit it doesn't happen all the time, but it does happen.

If you see this happen at all please post. Thank you.

[a_corpse_02]

I got a couple unknowns last night in Skyfire Mountains helping a cleric friend farm pearl frags for about 4 hours. But, we were slaughtering mobs by the truckloads so there would be a lot of repops and I only got one or two unknowns. I didn't think much of it.

[Dark]

My unknown spawns: Plane of Nightmares

Player::zoneBegin(): Pos (1668.000000/282.000000/212.000000) Heading: 5.000000
EQPacket: SEQClosing detected, awaiting next zone session, pcap filter: EQ Client 192.168.1.155
EQPacketFragmentSequence::addFragment: WARNING OpCode 0x6951 will not be processed due to loss
EQPacketFragmentSequence::addFragment(): recieved Out-Of-Order fragment seq 0x0006 (0x0002) expected 0x0000


I'm still getting some spawns but not many.

[cheese_poker]

An easy way to recreate an unknown spawn is to make a horse, and click it off. FYI.

[sea4th]

Unknown spawn behaviour described by Al is also observed in PoStorms but it is erractic. When there are very few peeps in zone or I am the only one hunting forest frogs the new spawns work fine. The spawn points light up and new spawns are decoded properly. When I first noticed unknown spawns or new spawns not being decoded it was right after the 4.3.18 release had been compiled from a fresh download and thought maybe the prior patched 4.3.17 did not exhibit this behaviour. After reinstalling the 4.3.17patched executable, the undecoded spawns are present.

In checking the spawn logs I have the following observation. I am soloing the frogs and someone else pops into the zone and takes a frog - after this frog is killed and they leave the next spawns were not decoded. I also noticed that the spawn log does not connect my player ID to the kills I make but does connect other players and my pet.

One last wrinkle - looking at the spawn list window shows unknown spawns are occuring even when forest frogs are being decoded. Rezoning / camping clears up the unknowns but loses the spawn point timer.

If there are other things or items I can watch for or test out please reply to this thread. With a spawn timer of 20mins it takes an hour to detect when unknown spawns are occurring.


Suggestions?

[Cryonic]

record a log of the packets and then see if Zaphod or another active dev can run through it to see why it is starting to do that. If it is showing things like:
Player::zoneBegin(): Pos (1668.000000/282.000000/212.000000) Heading: 5.000000
EQPacket: SEQClosing detected, awaiting next zone session, pcap filter: EQ Client 192.168.1.155
EQPacketFragmentSequence::addFragment: WARNING OpCode 0x6951 will not be processed due to loss
EQPacketFragmentSequence::addFragment(): recieved Out-Of-Order fragment seq 0x0006 (0x0002) expected 0x0000

Then maybe set Network -> Advanced -> Arq Seq Give Up to something higher than 256 (512 or maybe 1024).

It is possible that if packets are getting fragmented, then SEQ may be missing pieces coming over the wires and so once it gets an unknown mob it can't decode any after it (something to do with the way encoding is done now).

[tamasine]

I'm getting this error since upgrading to DSL - but only when running down/uploads from the same computer as everquest.. i thought it might be the network card - but it did occur to me that if showeq is capturing all packets from a host - then it might get overloaded.

t.

[SeqTester]

I do get unknowns also now and then. Not a big deal.. more wanted to comment on the "SEQ seeing other traffic"

I have found if SEQ is running when I remote desktop another machine from my EQ box it will crash. And EQ isn’t even running.

So yes it does see other packets, that’s what a packetsniffer does.

ST

[tamasine]

the pcap libraries have the ability to filter packets. if seq is requesting a full snoop of another machine, i can see that if it does a large transfer in the background, then seq may get overloaded with data. if however the snoop is of udp, eq-likely ports then it won't. (hmm. anyone know if bittorrent uses udp or tcp? - that'd explain it if the filter is "udp". i should really go and hack that bit to add "and not port <insert port here>")

"that's what a packet sniffer doesn" is relatively unhelpful. you meant "that's what a promiscuous interface does" - a packet sniffer is configurable to what packets it'll notice... as demonstrated by the network option to only snoop packets from a specified ip address

[Cryonic]

Most file sharing protocols are of the TCP variety. As for EQ, I would think that it would have a little of both. TCP to maintain your presence in the game and to receive data when zoning, UDP to deal with movement and combat.

[Zaphod]

The EQ protocols that ShowEQ monitors are all UDP based and come in on a wide variety of ports. You can see the current pcap filter being used by ShowEQ at any time. Just bring up the "Network Diagnostics" window from the "View" menu. Then look under the "Pcap Thread" entry's "Filter" sub-entry.

As far as what else is UDP based, almost all audio/video conferencing and streaming protocols, some file sharing protocols, part of DNS, most network file system protocols, dhcp, and a wide array of others. I just looked at the bittorrent protocol specification and it is TCP based, so as long as the pcap filters aren't seriously malformed it shouldn't capture them.

Feel free to have a go at improving the filtering ShowEQ does.

Enjoy,
Zaphod (dohpaZ)

[tamasine]

hmm. i had a feeling it would be. which means the problem is likely to either be the hub, or that network card, or the seq computer being unable to keepup with a 10bT network at the same time as running X+seq.. which actually might be the case. (showeq on idle [no EQ running] is taking 40% cpu)

bleh. ;)

[tamasine]

hum. wierder and stranger said alice.

if i kill all the torrents, nothing changes. restart seq - and it's back to its normal damn-good self.

puzzling.. i'll keep poking.
(and 40% cpu was with no eq running. start eq up and showeq drops to 10% ;)

[Cryonic]

Wonder if SEQ is getting up in a loop dealing with packets when EQ isn't going. I leave SEQ running when EQ is off and don't see that kind of problem. I run it on a P150 laptop and a Quad-PPro 200 box. System load on both is 0.00 when EQ isn't up. I do have things like BT going on in the background whether I'm in EQ or not on several systems, same with Kazaa at times.

[Alfred]

I have to believe it is a sequence of unique opcodes. Similar to what happened (i'm assuming) the last time.

I'm seeing it happen again in HoH right now.

[tamasine]

ok - it's nothing to do with high volume/overloading. i'm still getting unknowns with nothing but everquest on the network.. and seemingly in all zones - though a random time before it looses synch ;(

[Dedpoet]

Any chance you guys with the problem are having the PC go to suspend mode?

I only bring this up because I run seq on a laptop. The BIOS is set up to go to low power only if the AC power is not plugged in. Last night, I bumped the laptop with my elbow and didn't notice I knocked the AC adaptor free. A few minutes later the screen blanked. I realized what had happened and plugged the adaptor back in, but from then on I had unknown spawns.

It's the first time I ever had the problem with .18 and once I zoned I didn't see it again all night. Maybe the PC is going into an idle mode and the nic is getting powered off briefly and misses some packets. I'm not a harware expert at that level, but something to consider anyway.

[tamasine]

hrm... i think i might notice suspending the pc and back. no, it's not that. (desktop, power mgmt disabled)

problem still exists in seq5

[Zaphod]

If it is only 1 to 4 unknowns I wouldn't worry too much. If, however, ShowEQ is losing sync for some reason that could be a real problem.

First, try increasing the "Arq Seq Give Up" parameter under "Network" --> "Advanced" to 384 or 512 and see if that improves the behaviour. If it does, great, it just means that the EQ servers are having to retransmit packets more frequently to you

If that doesn't improve the behaviour, and you don't mind surrendering some level of anonymity to me you can e-mail me a packet recording that exhibits the problem.

To do this you would use the '--record-filename=<FILENAME>' command line option when launching ShowEQ. Then you should go through a session and attempt to reproduce the problem. If you are able to reproduce the problem, stop the EQ session, compress the recorded packet stream (preferably with bzip2) and e-mail it to me at [email protected]. At which point I should be able to reproduce the problem.

All information in the packet recording will be maintained in the strictest confidence.

This was how we resolved the last problem that caused ShowEQ to lose sync. The last time we were having a problem it turned out to have been caused by a previously uknown bit in the packet header being set and ShowEQ not knowing how to handle it.

Thanks and Enjoy,
Zaphod (dohpaZ)

[tamasine]

arq seq giveup set to 1024 .. still looses it. it's veery odd.. i've had a synch fail before i finished zoning (!) - to the degree that seq4 had missed part of the new-zone info (new map loaded, all spawnpoints as for last zone). seq5 seemed to be winning 'till it lost it 20 mins in.

i'll try to find a damn-fast(tm) zone that looses it and record

[tamasine]

is there a way to turn recording on while seq is running?
i ask because the problem may be connected with seq runtime:
at the point where every zone produces unknowns, and zoning to BB nothing decoded. i restarted seq in record mode.. and everything worked.

failing that i'll just have to try a low-traffic zone and hope the log isn't too big.

with seq5 i'm seeing fewer "synch lost" messages - more it just fails silently, chat output drops to only what i emit rahter than everything, and in the worst cases it doesn't see zoning at all. (crossing PoT with seq showing PoI + smartie spawns)

[reaver]

I was doing some searches of a problem I was noticing with 5.0.0.3 last night and came up with this thread seems to be the closest related.

I haven't seen any Unknown Spawns, but it seems that some items, notably horses and player corpses do not get removed from the list/map when they are removed from the game.

Meaning, if you summon a horse, then click it off and move, it still shows your mount being left behind and there even tho physically there is nothing there.

This also happened with player corpses (not NPC corpses). If someone died and got rezzed, looted their corpse, it would still show the corpse on the map/spawn list.

Is this just a matter of me increasing my ARQ (missing packets), or is this more of a case of a bug that spawns aren't being cleared when receiving the despawn packet?

Thanks in advance. If you want me to do any --record-filename capturing or testing let me know.

-reaver

[tamasine]

i'm getting everything from "zone - 10 spawns.. all unknown" to "zone, all spawns there. move around. a few minutes later, corpses don't happen - mobs just stay there, unknowns appear" i've changed network hardware (no change at all), and sent a record to zaphod of a mild case.

hmm.. at the moment i'm in PoT with no skittles at all - only 41 unknowns (64 players in zone). the spawnpoints are for PoK but map is PoT. it's all a bit annoying since so few people have this problem :-(

for name tracking, seq is only useable for me on zone entry.. most of the time.

[S_B_R]

Sounds like Packet Loss to me, or perhaps too much extreneous traffic on your network.

[Cryonic]

or maybe not all the opcodes are correct yet. I see mana movement in the console, but no longer see hp movement (or even total hp any longer).

[tamasine]

Originally posted by S_B_R
Sounds like Packet Loss to me, or perhaps too much extreneous traffic on your network.

wish it was. 512kb ADSL with no traffic other than EQ - and it's just as bad as the same network with 50k down, 20k up bittorrent running.

[S_B_R]

Originally posted by tamasine
wish it was. 512kb ADSL with no traffic other than EQ - and it's just as bad as the same network with 50k down, 20k up bittorrent running. I wasn't really refering to your Internet connection. I was more refering to your internal network, perhaps if you could describe your network Topology it may provide some insight.

One thing you could try is running some simple pings between your EQ machine and you SEQ Machine. In windows
ping -t 1.1.1.1Where 1.1.1.1 is the IP of your linux box. In Linux
ping 2.2.2.2Where 2.2.2.2 is the IP of your windows box. CRTL-C to break out of both of those. See if there are any dropped packets on either system.

Next thing to try would be ping a box on the internet from your EQ machine.
ping -t 64.37.156.21Then while that's running go to your SEQ box and do
tcpdump -i eth0 -n | grep 64.37.156.21 | grep icmpWhile both of those are running you should see 2 messages on your linux box for every 1 message on your windows box.

64.37.156.21 is patch.station.sony.com btw.

[SeqTester]

If this was already posted I am sorry. I did not read all the previous posts.

I saw someone talking about synch fail, a good way to tell if it is this or just a "Bad Packet" is after you get a "synch fail" and you Buff yourself it shouldn’t show the Buffs... Actually shouldn’t show ANY spells your casting.


ST.

[tamasine]

Originally posted by S_B_R
I wasn't really refering to your Internet connection. I was more refering to your internal network, perhaps if you could describe your network Topology it may provide some insight.

i think you missed my point. that's my connection speed, and eq is all that's on the network. running tcpdump on the ethernet shows *nothing* but EQ (no ssh - on console). equally, if it were dropped packets then saturating the link should make it worse.. nothing changes.

it's ADSL -> NAT router -> ethernet hub (tried two) with seq on a seperate box. having had and tracked connection problems in the past ('02) when my ISP instituted anti-DOS code which had the lovely effect of dropping eq packets, i've seen eq packet traces before - and these look perfectly normal.

i did all that pinging just for kicks, and got no dropped packets anywhere. honestly. the network works.. and this problem started after a patch (or a seq update, but they're obviously slightly in tandem)

[S_B_R]

The reason I pointed to your network is because you seem to be the only one with this problem (possibly Alfred too). I don't have it, none of the people I play with that also use SEQ have this problem. So logically something with your particluar setup is the culprit...

The ping things were just a stab in the dark. You'd have to let them run for q few minutes atleast to have a chance at seeing something I bet. Granted EQ isn't a bandwidth hog, but it surely takes alot more than a simple ping, which would make ping less useful...

[BlueAdept]

I get it too sometimes.

When it happens, it totally stops decoding new spawns and wont update old ones. If it is missing a packet, why does it just stop decoding? Is it the way the decryption is done that it cant skip any packets?

[S_B_R]

I'm 99% sure that's the case BA.

[tamasine]

ping: 3800packets, 0 packets lost

ping -s 1000: 266 packets, 0 lost (stopped that due to kiting)
ping -f: 147853 packets , 0 packet loss

i realise that this network is most likely the problem, but i really can't find anything. the alternative is that it's something to do with my account/characters.

[Alfred]

I don't believe it is a dropped packet. I do believe it is simply a different opcode sequence that seq doesn't know how to handle. Why?

This happened a few weeks ago with the exact same results and that is what the verdict was. Does it happen alot? No it doesn't, and therefore I've been lazy about sending in a capture file to nail down the problem. :)

[tamasine]

i'm afraid not alfred - i tried it from work.. worked fine. i rebuild seq on my desperately old machine (p166 is *not* a good idea ;) and it works fine.

so it's not the network - it would seem to be network card related.

[Alfred]

Perhaps we are talking about different things?

How could, testing at work really resolve the issue? In my case there isn't a magic thing to do or wait for. The problem is.. that the unknown mobs are excactly that, unknown. How can you "know" that it works 'at work' when it is more likely that you simply haven't seen the unhandled sequence yet?

[tamasine]

maybe it was different. i got consistent total unknowns. i know this because i could zone, and get unknowns before zone finishes.. staying in a zone for more than 5 minutes caused everything new to go unknown. i sat in a high traffic area (PoK) for an hour - which would have been unthinkable on my home computer to still get decodes.

basically the problems i was having were such that seq could only track the zone-spawns .. and then only intermitantly after a couple of minutes. any new spawns were right out.


so - being able to wait on a quest mob spawning for 2 hours was a pretty good test. and it worked. i think installed it on the crappy old computer at home. that worked as well. (and still is).

i can still not find any problem with the original machine/network card - but another machine on the same hub is working fine where that one was unusable.

so my problem - with unknowns *was* due to a faulty computer in some form and not the network/hub/EQ.

if you're getting occasional unknowns - fine. get a packet log and send it to zaphod. i did. but the packet log i sent was 1 minute long, and included 20+ unknowns zoning into/out of PoT. (i targetted some, and they wouldn't target until the spawn moved in seq)


and now i'll be trying 5.0.0.4 to see what it does on my frowned-upon machine.

[balmerch]

I am running the beta on a linux box with Fedora Core 1. Upon zoning, sometimes it won't load anything, or something it will load the map and no spawns, or sometimes some spawns too but not all, and other times everything. If it does load everything, after so long, it stops functioning completely... or partially. I will be picking up drops on the ground and they won't disappear, npcs dont turn to corpses, corpses don't disappear, spells don't update, movement isn't updated (except mine), and unknowns start popping up all the time. Now here is the best part, I have a laptop that I have run seq on for a long time, if I use this with the 4.x it works fabulously! I haven't put beta on the laptop yet, and I haven't put 4.x on the regular linux box yet, waiting for the 4.x update for that. But just to say, its probably not the network causing it since I have two computers and only one does it. When the 4.x update is released ill say if that fixed my problem or not.

Balmer

[Smoker]

Since the day GoD came out I have had the problem of Unknown Spawns. The day(s) before that, it was working fine.

I am using the latest release of SEQ 4x.

Recompiled with the new decode.cpp that was posted (which stopped many seg faults after GoD came out).

What happens for me is:

Zone into any zone, everything is fine, map and all spawns.

Start killing stuff, you can see the mob's corpse and the name of the corpse. Then respawn on all dead mobs are 'unknown'. Any players that zone in after me are 'unknown'. Also if a mob spawn is triggered from killing other mobs, the new spawned mob is unknown.

If I zone out then in, or die and get rezzed in, SEQ decodes all just fine.

I got no clue what this could be, just what others have said, maybe respawn comes with a new opcode?

[Cryonic]

Last I checked, SEQ 4.x is not fully fixed to work with GoD. The patched decode.cpp helps to fix some of the issues, but not all.

[Smoker]

It would be interesting to know if anyone does not have this problem at all with SEQ 4x.

[Cryonic]

Judging by the lack of response I would have to guess that quite a few people don't have this issue with 4.x or 5.x.

That 5.x on one box has issues and 4.x on another had no issues doesn't rule out the network, it just rules out any pieces of equipment that are the same between them (e.g. the router, hub, etc...) It does not rule out issues with a network card, cables, etc...

[balmerch]

I tested out the 4.x version, and had the same problem on my linux box. So it works on the laptop but not on the tower, they use the same crappy hub, now I will be changing the network card and network cable. I will post back when I can with results.

Chris

[Velium_hound]

Originally posted by Cryonic
Judging by the lack of response I would have to guess that quite a few people don't have this issue with 4.x or 5.x.


Actually, I do see this problem (4.x) but am assuming it has to do with the GoD opcodes. Since work is in progress already on that issue (another thread), I haven't posted an "I'm broken too" since it really accomplishes nothing at this point.

Woof