Help Decoding Datastream?

[AMadMonk]

I can post logs that show EQ's in-game text and decoded packets (byte/ascii).

If I do this, is anyone willing to help me start to decipher the protocol?

[Cryonic]

Join IRC channel, find ksmith or Zaphod (maybe even basic) and see where it goes from there. Posting whole logs to the Net would reveal char name and other data that might be better left shared with as few people as possible.

[AMadMonk]

Yeah I realized after I posted that this afternoon that my toon's name was *everywhere* in that log.

I'll join the channel. I bet the protocols are similar enough that an EQ2 version of SEQ should be pretty easy.

[Cryonic]

IRC is still pretty quiet.. but seems like a number of individuals have come and gone from the channel wanting to work on SEQ2.

[Rohan]

I'm interested in helping with deciphering the packets. I have been isolating the packets too and digging in the program for the last few days, then I found this thread. I will be hanging around in the irc channel the next few days.

[Alethal]

If you post logs here or leave me a private message with those, I will work on them.

[AMadMonk]

I have a decent-sized packet log, but I want to go through it and scrub out my toon's name before I share it out. That's not 100% "protection" against it getting into the wrong hands and me getting banned, but then again, there's no such thing as 100% security.

The packet log I have is actually debugger spew that shows EQ chat log lines interspersed with hex dumps of packet contents. The packets have been XOR-decrypted and, where necessary, inflated.

I'll have to go through this 2meg file with a fine-toothed-comb before I post it, so gimme a day or two.

[Druppy]

Hey,
I'm on irc most of the time now waiting for some set of you guys to show up so we can start cracking on this. I'll be on as Druppy.

[LordCrush]

Reads cool

i wish you guys good luck and offer help with beta-testing *fg*

[AMadMonk]

Incidentally, what I was calling the "xor encryption" is in fact an RC4 stream cypher being performed on all inbound and outbound packets. I've reversed this encryption for all packets (with an injected, detoured DLL) and learned where in memory the keys are being kept (they are offsets into an object; everything in EQ2 is C++, so say goodbye to the days of plentiful and useful global variables).

About 1% of the packets coming in are compressed with zlib. I haven't seen any outbound packets compressed (which makes sense). It took me a bit of futzing to get the zlib stuff to work on the inbound packets because they're inflating the compressed inbound packets as one continuous stream rather than as one discrete inflate per packet (which makes sense, since this allows them to build up a bigger dictionary -- is that the right term? -- and theoretically, over time, get better compression). So I had to copy the state of their zstream object prior to doing the inflate on the packets tagged for zlib work, before I could make the compression stuff work properly.

One nice benefit of doing stuff with an injected DLL (rather than just looking at the packet stream in situ) is that I have access to the class data itself. It appears that the packets are just serialized instances of in-memory objects, and there's one common base to all the different types of network packets. The benefit of *this* is that now that I've finally reversed about 90% of how the serialization code works, I can use the RTTI info in the classes to get human-readable names for each object type. In other words, I have access to the class names that Sony itself is using, for each opcode. So, instead of just saying this is opcode XXXX and I think it does this, I can say "this is opcode XXXX and it's a VeRemoteCmdMsg" or whatever.

I'm a bit leery about posting logs, still, since even if I scrub my toon's name out of the data, there's probably still enough info there to get me banned... What to do, what to do...

[AMadMonk]

Further info, when you zone, a lot of interesting data appears to be sent. I don't know how this was with EQ1, but it looks like a full spawnlist gets sent, along with descriptions of every ability (spell, combat art, etc.) you have, every recipe you know, and so on. Also, your entire quest journal status appears to be sent.

Interesting stuff.

[AMadMonk]

Last but not least:

I got sort of burnt about publishing my code in EQ1, so I'm probably not going to post my full detours code (here, or anywhere). However, I'm perfectly willing to code up a trimmed-down RC4 key extractor to make the process of others decoding the datastream easier.

Would that be of use?

[Rohan]

Yes that would help a lot. Thanks to your help I'm atleast able to watch the programs packets after decryption but being able to exctract the key smoothly and decrypt the packets oneself would be a lot nicer

[terminate]

I am very interested in working on this project. I will be hanging out in the IRC room in the free time. Looking forward to getting started.

-E

[terminate]

Sorry for the lacking post above, had to run get ready for work.

I am a C/C++ programmer with some knowledge of Java. I have worked on both Linux and win32 platforms, but feel more comforatable on win32. I am of course also an avid gamer, playing a multitude of games, including EQ2.

I admittedly have not done a lot of network programming or protocol reverse engineering, but look forward to improving my skills and using my other knowledge to help furthur the project.

I will be hanging out in the IRC room as I have free time (probally during work as well, although I will be mostly AFK during that time trying to get actual work done).

I would appreciate if someone would contact me and get me started with the decrypting of the packets. I have set up a logger and am ready to capture, decode, and analyze the data in an attempt to decipher EQ2's data stream.

I look forward working on this project and cannot wait to get to work.

-E