EQ2 Packet Sniffer

[Acid1789]

Hi,

For those of you who dont know, I have been working on figuring out the eq2 packet stream for some time.

I have a working eq2 packet sniffer. It decrypts and decompresses as well as identifies every packet.

You can download the source to it here: http://www.pptournament.m6.net/eq2

[LordCrush]

Wow thank you :)

[int 21h]

I've been playing with this a bit, but I never recieve many packets. The last packet I recieve is the unimplemented opcode 5 in loginstream.txt. Is it the same for others? After that I no longer have any other packets logged...

[int 21h]

Actually, it looks like the decryption key isn't being returned by getEQ2key()

....

I imagine the key moves with patching of the game executable, when I look at that location in memory I see a bunch of the crypto functions, but I am not sure which key I am looking for. Any hints on this Acid?

[Alethal]

Hi,

Indeed, the key moves each time the game is patched. I have not looked into this deeply but here is what I *think* I know:

- Like MadMonkey said, the key is an offset into an object. EQ2GetKey knows where in memory the pointer to the object will be stored, then it adds an offset (0x60) and grabs the key from there. Each time the game is patched, the area where the pointer is stored changes.

- There is an easy way to find where that area is. The point is only to find again the function that reads/stores the pointer. I only looked into this for 30 minutes yesterday but basically, it's the function in which there is a call to _srand somewhere. The offest is before all the "mov esi[...] eax[...]" calls. It should not be difficult to have the program automatically detects the new area. (check the version of the game, check the version stored, look for a pattern of bytes - which, hopefully, will be unique-, deduces the key position)

Right now my concern is more to know how Acid found this. I tried starting from the recvfrom() and sendto() calls and make my way slowly to the function that decrypts/encrypts the stream but it's way too tedious... :p

[Acid1789]

I released a new version with updated information. Should work with the latest patch.

Thanks again to those who have donated.

http://pptournament.m6.net/eq2

[Acid1789]

Yet another update.

Adjusted key offset to work with the latest eq2 client.
Added new packets to match eq2 client.
Restructured the code to be more modular and easier to modify.
Added new SSE2 brute force algorithm to break keys(currently disabled)
Fixed a few instability issues.

http://www.pptournament.m6.net/eq2

[Acid1789]

New version, for anyone who is interested :)

Adjusted key offset to work with latest update
Fixed a few instability bugs.
Added support for zoning!

http://www.pptournament.m6.net/eq2

[Megaton]

Nice work! :-)

I wish I had more time to try and figure out the structures, there are SO many good uses for a decrypted data stream :-)

-
Megaton

[Alethal]

By the way, EQ2 does use a "bubble"-like system for spawns. You only have the list of the mobs that are in a certain radius from you. I am not sure about harvest nodes, someone was saying it might send all the nodes in the zone.

What is interesting is that there is a way to know in-game whether a mob is up or not (similar to the trick used by mages in EQ1), I wonder to what extent that could be used to populate a full list of mobs in the zone.

[Acid1789]

Yes, EQ2 does only send ghost info for a small area around your character. This includes harvest nodes, ?s, widgets, npcs, monsters, players, etc.What trick are you refering to in eq1?

[int 21h]

Tracking extends out of the bubble, how is it implemented?

[Acid1789]

Im not really sure how tracking works (havnt played a tracker class). But from what I have seen, my guess is that it doesnt extend outside of the bubble.

Trackers dont seem to be able to track mobs that are all that far away (1000m or so).

[Megaton]

Hey Acid1789, would you mind explaining a bit about how this all works? Like how channels / streams fit together, and also where XOR Compression fits in?

[Acid1789]

Yeah sure,

A stream is a 'connection' between the client and the server. Each stream can have up to 4 communication channels. A channel contains sequenced packets. This is all independant of the actual game data, EQ1 shares the same protocol.

EQ2 is only using 1 channel and mostly only 1 stream at a time. Every time you zone, the client connects to a new zone server and creates a new stream. All eq2 messages are on channel 0. Due to their encryption implementation, they are incapable of using the other channels unless they changed the way encryption is done.

Channel packets are all sequenced. Fragmented packets are reconstructed in order. A fragmented packet is basicly one large packet spread over smaller sequenced packets. Once a packet is complete, it is decrypted using the RC4 cipher. If the packet is compressed, it is decompressed with a single call to inflate(zlib). The packet is then processed by the packet handler.

Some eq2 packets have Compressed XOR buffers. This is a simple delta compression scheme. The way it works is, if you have a buffer you are repeatedly sending (position info), you XOR the current data with the last data. This turns all the bytes that are the same into 0s leaving you with only bytes of data that are different. Then the buffer is compressed using a byte level RLE scheme to 'remove' all the zeros. The decompress is the inverse, expand then xor with the previous data.

[int 21h]

I've tracked some pretty long distances before and it seemed to go well beyond LoS

[Megaton]

Thanks for the overview, Acid.

So with the XOR compression you will need to maintain state throughout the stream to decompress these, right?

How do you know which sections of the packets has these XOR buffers, and how many of these are there? For example, is there one for prediction updates, one for ghost creation, etc?

And one more question, is XOR compression responsible for the mangling in the npc names in Ghost Creation:
<pre>
0140: 44 65 ..............De
0150: ff 70 75 74 79 20 43 6c ff 69 66 66 6f 72 64 73 .puty Cl.iffords
0160: 83 6f 6e .on
</pre>

[Acid1789]

They are using the XOR delta compression all over for alot of things. Identifying it is part of reversing any of these packets. And yes that is why the names of ghosts are mangled. Here is the unpack for the ghost packet from EverSpy2




void CGhost::Create(BYTE* pCreateCommandData)
{


DWORD dwBytes = ReadSwappedLengthFromBuffer(pCreateCommandData, (int*)&m_wGhostID);

m_dwEntityID = *(DWORD*)&pCreateCommandData[dwBytes];

WORD wUnkValue = *(WORD*)&pCreateCommandData[dwBytes + 4];

BYTE byVerbCount = pCreateCommandData[dwBytes + 6];

dwBytes += 7;

char szTemp[512];

if( byVerbCount )

m_pVerbs = new FXString[byVerbCount];

for( BYTE i = 0; i < byVerbCount; i++ )
{


WORD wVerbLen = *(WORD*)&pCreateCommandData[dwBytes];

dwBytes += 8 + wVerbLen;

wVerbLen = *(WORD*)&pCreateCommandData[dwBytes];

memcpy(szTemp, &pCreateCommandData[dwBytes + 2], wVerbLen);

szTemp[wVerbLen] = 0;

dwBytes += 2 + wVerbLen;

m_pVerbs[i] = szTemp;

}

WORD wDefVerbLen = *(WORD*)&pCreateCommandData[dwBytes];

if( wDefVerbLen )
{


memcpy(szTemp, &pCreateCommandData[dwBytes + 2], wDefVerbLen);

szTemp[wDefVerbLen] = 0;

m_szDefaultVerb = szTemp;

dwBytes += wDefVerbLen;
}

dwBytes += 2;

m_fDefaultVerbRange = *(float*)&pCreateCommandData[dwBytes];

m_dwNumEncounterMembers = pCreateCommandData[dwBytes + 4];

dwBytes += 5;

if( m_dwNumEncounterMembers )
{


m_pEnounterMembers = (DWORD*)malloc(sizeof(DWORD) * m_dwNumEncounterMembers);

for( DWORD i = 0; i < m_dwNumEncounterMembers; i++ )
{


m_pEnounterMembers[i] = *(DWORD*)&pCreateCommandData[dwBytes];

dwBytes += 4;

}
}

m_dwServerTime = *(DWORD*)&pCreateCommandData[dwBytes];

DWORD dwDataLength = *(DWORD*)&pCreateCommandData[dwBytes + 4];

if( dwDataLength )
{

BYTE byDecompressed[878];

DWORD dwOutLen;

CompressedXORDecompress((BYTE*)&m_sGhostDelta, 878, &pCreateCommandData[dwBytes + 8], byDecompressed, &dwOutLen);

memcpy(&m_sGhostDelta, byDecompressed, 878);

}

}

[Megaton]

Everspy2? What is that? :)

[Acid1789]

EverSpy2 is the new tool ive been working on. Its a map display similar to showeq :)

[int 21h]

What ever happened to EverSpy1?

[Acid1789]

I stopped supporting the public version of EverSpy1 due to the ammount of complaints I got. It was originally an open source windows clone of showeq. It was met with harsh resistance from the showeq people who believed at the time linux was the only true god and anything else was inferior. Also, no one else wanted to help with its development, they only wanted to bitch about features that werent in it.

I continued to support it for myself and a few friends, but never again released a public version. Eventually I stopped supporting it, and stopped playing EQ1 all together.

[SEQSniffer]

People usually get a lot more grief than they do praise, even when doing a great job...I, for one, want to thank you for taking the time to work on this project. Many of us kept quiet for years & sat in the background while enjoying SEQ....Keep up the great work, we truly appreciate it. I would offer to lend my services with programming; however, I am clueless when it comes to that department ;). I'll cheer you all on quietly from the sidelines :p.

[RavenCT]

I a Windows NT MCSE, a Windows 2k MCSE and a Windows 2k MCSA, and I can say that I greatly appreciate those out there that write, support, debug, and somehow make all that magic work that made ShowEQ and any other product or work that "assisted" with the original EQ and hopefully at some point EQ2. Being that I come from the network/hardware side and not the development side, I wish I could help, but I lack the tools and knowlege. I thank everyone that has developed these apps and hope you all don't loose faith and know that there are us out there that appreciate the work!

[BlueAdept]

I stopped supporting the public version of EverSpy1 due to the ammount of complaints I got.

I didn't realize that was you Acid. I remember you from way back on the HQ boards.

I still have your bookmark from long ago.

http://www.geocities.com/winseq/

also

http://everspy.sourceforge.net/

But it didn't look like anything was ever done with that one.

WB to the EQ(2) scene.

[Acid1789]

Heh, thanks for the ecouragement.

Heres a screenshot of EverSpy2 in its current state.

http://www.pptournament.m6.net/eq2/EverSpySS.jpg

[Megaton]

Nice screenshot, looks like it is coming along nicely!

I know there was quite a bit of animosity between the linux and windows community. I believe part of it was that the linux community believed that if ShowEQ crossed over to Windows it would increase the user base to such an extent that Sony/Verant will take much more notice and do something drastic.

Shortly after I released the ShowEQ source to the public I started losing interest, mainly due to the large amount of time required to work on such a project :-). But the open source community was quick to take over. It was amazing to see such dedication. I enjoyed using ShowEQ for quite a while after I stopped developing on it due to their efforts.

Yours and other's work is definately apprieciated :-).

I stopped playing EQ1 a long time ago, but EQ2 and the source you posted of your EQ2 packet sniffer has me hooked again ;-). I even installed Microsoft Visual Studio! /gasp!

-
Megaton

[int 21h]

Are those mobs and ?s showing up just within your LoS in that screenshot? Does zone geometry impact it at all? (Walls, floors, etc.)

Have you made any decisions on how/if you'll release your program? Should people in the community that want to contribute to an effort like this just start trying to build something or will there be an opportunity to assist you?

[int 21h]

Also, for those of us who didn't really participate in ShowEQ or EQ1, can you explain the terminology a bit?

Ghosts are mobs and npcs? So a ghost create command is sent everytime an NPC needs to appear on the client screen? Is that what you're parsing for Everspy2?

Thanks in advance for any hints.

[Megaton]

I dont think LoS has anything to do with it, but I might be wrong. There is a radius around you in which you get updates for. This radius is most likely larger than your LoS, since you can actually adjust that in-game with the Model Detail, Leve of Detail Bias slider.

Some other ghosts are sent to you regardless of range, for example your group mates.

[Acid1789]

Ghosts are mobs and npcs? Mob = Mobile Object
Ghost = EverQuest2 term for client side representation of a Mob

Ghosts include NPCs, Players, ?s, and Resource nodes. There is a separate grouping called SignWidget for things like mariners bells, shop signs, etc.


So a ghost create command is sent everytime an NPC needs to appear on the client screen?
Yes, whenever any ghost comes into range the server sends an EqCreateGhostCmd packet to the client. While the ghost is within range of the player, the server sends EqUpdateGhostCmd packets for that ghost. When it leaves your little area, the server sends an EqDestroyGhostCmd packet.

I dont think that zone geometry impacts it at all, but It does appear to only give you new ghosts within your view frustum. Im not 100% on this, just looks that way from what I have seen.


Some other ghosts are sent to you regardless of range, for example your group mates
This is accurate, you will get ghost info for anyone in your group regardless of where they are in the zone.

[Megaton]

Ghosts include NPCs, Players, ?s, and Resource nodes.


Crafting stations are also ghosts - they get created with the EqCreateGhostCmd command. Only reason I know this is because I've been working on the tradeskill packets a bit in the tradeskill instances :-).

[Freakyuno]

but It does appear to only give you new ghosts within your view frustum.

This is very possible, I believe a while back the original EQ implemented something they were calling the Umbra Engine, which was a D3D command set that allowed for only rendering things in your view, instead of having to render everything when the zone loaded.

This same type of technology can be used to detect 3d position and direction and determine the need to create ghosts, or not.

[Acid1789]

Rendering is one thing (and they would be insane to not be doing frustum culling). But loading is another issue.

Doesnt make sence for them to only load whats in view. You wouldnt know about anything that came up behind you. So if you spun around, it would stall while loading the new mobs.

After looking at it a bit closer, I think it may be some sort of occlusion culling with a colission mesh. But thats still just a guess.

[akanon]

I for one appreciate all of the work that you are putting into EQ2 tools. Just a suggestion for ya - if you wanted, you could use the maps from maps.eq2interface.com as a backdrop for EverSpy2. It would probably make it a bit more useful for you and your buds - just my 2 cents.
Oh - and sorry for the e-mails I sent ya before - I misread something on your website.

[Acid1789]

I put up a new version of EverDump. Works with the latest patch. Also I added code to grab the key address from the exe every time you start everdump. It now works if you start from the launcher. Also works if you start it after eq2.

Beta for EverSpy2 has begun :) All the people who have donated to my project should have gotten a copy. Please let me know if I missed anyone (and yes I keep records).

http://www.pptournament.m6.net/eq2

[Acid1789]

Posted an updated version of EverDump to work with todays patch (if anyone still cares). There are a few other minor changes to the stability of the SonyProtocolLib as well.

There is a new EverSpy2 release available too for those on the beta list (check your mailbox).

http://www.pptournament.m6.net/eq2

[Choofer]

Excellent, looking at my email now.... hmmm nothing but thats ok im sure it will come soon


Choof

[Freakyuno]

<- checks email...nothing yet. ;)

[Acid1789]

It was brought to my attention today that the last two releases of EverDump didnt have the SonyProtocolLib source in them. I put a new version up today that includes the SonyProtocolLib again.

Also a few people have asked me about specific packet structures. I havnt spent alot of time figuirng out individual packets yet, there are only a handful that Im using for EverSpy2. I will add some sort of documentation for the packets I have already reversed in the next couple of days. Hopefully, those of you who are intersted in packet formats will submit some of your findings :)

[jakep]

Will EverSpy2 eventually be available freely?

[Acid1789]

No, EverSpy2 is commercial software. It will be available for purchace and there will be an optional subscription fee for the community mode features.

[cafrow]

No, EverSpy2 is commercial software. It will be available for purchace and there will be an optional subscription fee for the community mode features.
You take credit cards? I want. :D If you need a beta tester let me know, also if you need a dumpsite, or download mirror let me know

[tonyis3l33t]

he has a paypal donation on that site he has up there.

[BSRdr]

he has a paypal donation on that site he has up there.
I did donate through that link yesterday, so hopefully I can try out the beta product :)

[taco-man]

when i try and run the exe file that is included in the 1.06 zip file it opens a command prompt window that is blank and then performs an illegal operation in about 5 seconds it if i debug it, it has "Unhandled exception at 0x10016a01 in EverDump.exe: 0xC0000005: Access violation writing location 0x00000001." when i compile it myself i get the same error and it occurs on this line:
if ( !ReadFile( (HANDLE)_osfhnd(fh), buffer, cnt, (LPDWORD)&os_read,
NULL ) ) {

any ideas? am i a complete idoit and overlooked something i dont know about.
/slap computer...multiple times.
when i reset it worked so you can disregard this post, whatever it was has corrected itself on both my compiled version and the one already included in the zip.

now that it is working it just sits there and says listening on network adapter on localhost and doesnt show anything once i login to a charater and into the zone. is it not compatiblew with the most recent patch?

[Acid1789]

Did you install winpcap?

[Choofer]

Just on that winpcap, what version should we be using i have installed the beta version, has that been tested as i am having problems maybe that is the reason


Choof

[Acid1789]

The version I am using is WinPcap 3.1 beta 4

[Choofer]

OK, thats not my problem then ill go back to the drawing board and try the suggestions you gave me in the e-mail

Choof

[Acid1789]

What problems are you having?

[Choofer]

You have posted an emails re: my problem

Not being able to select a lan card when EverSpy starts up, IP address is right, ping tests work, no firewall between the 2 machines. This is only when doing a remote connection. When i run it locally it works fine.


Choof

[Cryonic]

are you on a physical cable or wireless?

Most wireless cards drivers don't allow promisc mode for sniffing.

[Choofer]

Physical cable, it used to work from this computer but dies around 3 - 4 weeks ago and since we have been getting regular update's i have just been trying each new update, last weekedn i even rebuilt my pc and removed one of the lan cards just in case there was some sort of conflict.

The inability to select the type of lan interface is the part that bothers me i just dont even get asked it just pops straight into the options screen to select local or remote....

I'll try running esremote of the local machine to make sure there is data being sent by eq tonight and work backwards from there...


Choof

[Acid1789]

Im developing using a completely wireless network. Since ESRemote runs on the eq2 machine, there is no need for promisc mode (infact you cant set it to promisc mode).



The inability to select the type of lan interface is the part that bothers me i just dont even get asked it just pops straight into the options screen to select local or remote....

Does EverSpy2 crash/close after you select remote mode / local mode in the options dialog?

Can you see any network traffic between ES2 and ESRemote?

[Pasla]

Hi,
I'm working on a side project using the EQ2 packet sniffer. I'm interested only by recipes books, and I'm trying to decode the format of theses packets. This is what I have found:
- strings are stored in some BCPL format (not null terminated, but instead using one byte to give length then the string).
- i have not decrypted all the header of books packets, it seems to exists at least 2 types of packets. After header, offset 0x45 should be 0x00, then either you get 0x07 then the number of recipes in the book, or some "junk" (from 2 to 5 bytes), then 0x07 and number of recipes. I dont know why....
- each recipe have its length, then the string using UTF8 encoding (YES, very important).
- After all the recipes, one null byte 0x00 to terminate the list (useless but...), and you have the name of the book, still using BCPL format. Packet end with 0x00 0x00.

I'm trying now to decode the header of the packet, to see why some books have a different format than others. If you have valuable informations, please share :)

[Pasla]

To make the packet sniffer working on Test server, you need to add in SonyProtocol.cpp near line 326 :

+#ifdef TEST_SERVER
+ strcat(path, "\\TestServer\\EverQuest2.exe");
+#else
+ strcat(path, "\\EverQuest2.Exe");
+#endif
FILE* pFile = fopen(path, "rb");
if( pFile )