PDA

View Full Version : Help Decoding Datastream?



AMadMonk
12-01-2004, 07:39 PM
I can post logs that show EQ's in-game text and decoded packets (byte/ascii).

If I do this, is anyone willing to help me start to decipher the protocol?

Cryonic
12-02-2004, 12:27 AM
Join IRC channel, find ksmith or Zaphod (maybe even basic) and see where it goes from there. Posting whole logs to the Net would reveal char name and other data that might be better left shared with as few people as possible.

AMadMonk
12-02-2004, 02:21 AM
Yeah I realized after I posted that this afternoon that my toon's name was *everywhere* in that log.

I'll join the channel. I bet the protocols are similar enough that an EQ2 version of SEQ should be pretty easy.

Cryonic
12-12-2004, 04:20 PM
IRC is still pretty quiet.. but seems like a number of individuals have come and gone from the channel wanting to work on SEQ2.

Rohan
12-13-2004, 04:27 AM
I'm interested in helping with deciphering the packets. I have been isolating the packets too and digging in the program for the last few days, then I found this thread. I will be hanging around in the irc channel the next few days.

Alethal
12-13-2004, 05:53 PM
If you post logs here or leave me a private message with those, I will work on them.

AMadMonk
12-14-2004, 02:22 PM
I have a decent-sized packet log, but I want to go through it and scrub out my toon's name before I share it out. That's not 100% "protection" against it getting into the wrong hands and me getting banned, but then again, there's no such thing as 100% security.

The packet log I have is actually debugger spew that shows EQ chat log lines interspersed with hex dumps of packet contents. The packets have been XOR-decrypted and, where necessary, inflated.

I'll have to go through this 2meg file with a fine-toothed-comb before I post it, so gimme a day or two.

Druppy
12-17-2004, 07:55 PM
Hey,
I'm on irc most of the time now waiting for some set of you guys to show up so we can start cracking on this. I'll be on as Druppy.

LordCrush
12-18-2004, 08:42 AM
Reads cool :)

i wish you guys good luck and offer help with beta-testing *fg*

AMadMonk
12-21-2004, 01:39 PM
Incidentally, what I was calling the "xor encryption" is in fact an RC4 stream cypher being performed on all inbound and outbound packets. I've reversed this encryption for all packets (with an injected, detoured DLL) and learned where in memory the keys are being kept (they are offsets into an object; everything in EQ2 is C++, so say goodbye to the days of plentiful and useful global variables).

About 1% of the packets coming in are compressed with zlib. I haven't seen any outbound packets compressed (which makes sense). It took me a bit of futzing to get the zlib stuff to work on the inbound packets because they're inflating the compressed inbound packets as one continuous stream rather than as one discrete inflate per packet (which makes sense, since this allows them to build up a bigger dictionary -- is that the right term? -- and theoretically, over time, get better compression). So I had to copy the state of their zstream object prior to doing the inflate on the packets tagged for zlib work, before I could make the compression stuff work properly.

One nice benefit of doing stuff with an injected DLL (rather than just looking at the packet stream in situ) is that I have access to the class data itself. It appears that the packets are just serialized instances of in-memory objects, and there's one common base to all the different types of network packets. The benefit of *this* is that now that I've finally reversed about 90% of how the serialization code works, I can use the RTTI info in the classes to get human-readable names for each object type. In other words, I have access to the class names that Sony itself is using, for each opcode. So, instead of just saying this is opcode XXXX and I think it does this, I can say "this is opcode XXXX and it's a VeRemoteCmdMsg" or whatever.

I'm a bit leery about posting logs, still, since even if I scrub my toon's name out of the data, there's probably still enough info there to get me banned... What to do, what to do...

AMadMonk
12-21-2004, 01:42 PM
Further info, when you zone, a lot of interesting data appears to be sent. I don't know how this was with EQ1, but it looks like a full spawnlist gets sent, along with descriptions of every ability (spell, combat art, etc.) you have, every recipe you know, and so on. Also, your entire quest journal status appears to be sent.

Interesting stuff.

AMadMonk
12-21-2004, 01:44 PM
Last but not least:

I got sort of burnt about publishing my code in EQ1, so I'm probably not going to post my full detours code (here, or anywhere). However, I'm perfectly willing to code up a trimmed-down RC4 key extractor to make the process of others decoding the datastream easier.

Would that be of use?

Rohan
12-21-2004, 03:50 PM
Yes that would help a lot. Thanks to your help I'm atleast able to watch the programs packets after decryption but being able to exctract the key smoothly and decrypt the packets oneself would be a lot nicer

terminate
12-22-2004, 10:53 AM
I am very interested in working on this project. I will be hanging out in the IRC room in the free time. Looking forward to getting started.

-E

terminate
12-22-2004, 01:00 PM
Sorry for the lacking post above, had to run get ready for work.

I am a C/C++ programmer with some knowledge of Java. I have worked on both Linux and win32 platforms, but feel more comforatable on win32. I am of course also an avid gamer, playing a multitude of games, including EQ2.

I admittedly have not done a lot of network programming or protocol reverse engineering, but look forward to improving my skills and using my other knowledge to help furthur the project.

I will be hanging out in the IRC room as I have free time (probally during work as well, although I will be mostly AFK during that time trying to get actual work done).

I would appreciate if someone would contact me and get me started with the decrypting of the packets. I have set up a logger and am ready to capture, decode, and analyze the data in an attempt to decipher EQ2's data stream.

I look forward working on this project and cannot wait to get to work.

-E

Druppy
12-23-2004, 01:31 PM
Last but not least:

I got sort of burnt about publishing my code in EQ1, so I'm probably not going to post my full detours code (here, or anywhere). However, I'm perfectly willing to code up a trimmed-down RC4 key extractor to make the process of others decoding the datastream easier.

Would that be of use?
Not sure what happned with EQ1, but I'd like to chat you up about what you have and where you want to go with it, and what we can do as a group. (we = all the posters to this thread). I havent seen you on IRC recently, if you plan to get on toss me a PM or just email me [email protected].

~Druppy

Alethal
01-03-2005, 03:49 AM
What is EQ1 using for encryption? It was my understanding that a flaw in the EQ1 encryption scheme allowed us to decipher packets without grabbing the key from memory.

I guess that this will not be the case anymore with EQ2. We will need to get the RC4 key from the memory and then send it to the SEQ2 box. That might be a good time to develop the client/server application Zaphod mentioned someday.

Anyway, I did not understand the zlib part but I am not there yet.

LordCrush
01-03-2005, 12:30 PM
With EQ1 before POP ther was an encryption that could be borken with brute force attack. With the initial POP Release Sony changed it to something that could not any longer be brute forced. The "Vers 1" of this encryption had a weakness that could be used to reduce the range of possible keys to a range that was again practicable with brute force. This was worked about one week. Then they changed it to something that was *really* not more breakable with brute force. I think Zaphod wrote that days "They secured it decendly". That was the reason for the development of the keysniffers for seq. That was about October/November 2002. In February 2003 they changed the encryption back to a more ore less simple XOR that made the keysniffer unnecessary. I have no clue why they stepped back, but i assume that there were too much problems with the performance of many pcs and the harder encryption.

Sadly it looks like seq2 will probably not be possible without keysniffer :(

** PS.. i am not used to the encrypting terms, so excuse me for the not technical correct description :rolleyes:

BlueAdept
01-03-2005, 01:57 PM
I think Zaphod wrote that days "They secured it decendly". That was the reason for the development of the keysniffers for seq. That was about October/November 2002. In February 2003 they changed the encryption back to a more ore less simple XOR that made the keysniffer unnecessary. I have no clue why they stepped back, but i assume that there were too much problems with the performance of many pcs and the harder encryption.

I thought that someone just figured out the encryption. I didnt know that sony switched to back to the xor. I guess I should take a look at the libeq file some day.

Alethal
01-03-2005, 02:26 PM
Figuring the encryption would not help if they were using something like RC4. The only way to figure it without sniffing the key is to "brute force" it. I am not an expert but I think there is something with RC4 about being able to figure out the key faster if you know the first few bytes of the plain text.

Not sure if that would help at all anyway.

By the way... that might sound like a newbish question but I wonder something about EQ2/EQ1 encryption. Does it encrypt both the EQ Data HEADER and the EQ Data Payload or just the EQ Data Payload?

Looking at the ShowEQ source code, it looked like only the EQ Data payload was encrypted but I am not certain.

LordCrush
01-05-2005, 09:11 AM
I thought that someone just figured out the encryption. I didnt know that sony switched to back to the xor. I guess I should take a look at the libeq file some day.

Hmm i have looked through the forum yesterday, but i did not find the post yet. I think mvern posted it somtime, but it can been on irc sometime too.

Btw the fix was in april 2003 not Feb 8) - i have mixed up the dates :)

ksmith
01-05-2005, 01:49 PM
EQ1 went back to their zany XOR encryption quite some time ago. It basically just XORs 64 bits in each compressed payload, enough to make the decompress routines choke if you haven't been decrypting the stream up to that point.

From packetstream.cpp:

if (s_encrypt)
{
if (!m_validKey)
return NULL;

#ifdef PACKET_DECODE_DIAG
seqDebug("decoding 0x%04x with 0x%08llx on stream %s", opCode, m_decodeKey, EQStreamStr[m_streamid]);
#endif

int64_t offset = (m_decodeKey % 5) + 5;
*((int64_t *)(data+offset)) ^= m_decodeKey;
m_decodeKey ^= *((int64_t *)(data+offset));
m_decodeKey += *len;
}



I recall someone on irc mentioning that EQ2 uses RC4 from cryptlib.

Alethal
01-06-2005, 07:51 AM
cryptlib or cryptoAPI? Have not seen anything from cryptlib (no DLL called and nothing statically). I hope I have not missed the obvious :(

Dillemann
01-17-2005, 01:08 PM
Last but not least:

I got sort of burnt about publishing my code in EQ1, so I'm probably not going to post my full detours code (here, or anywhere). However, I'm perfectly willing to code up a trimmed-down RC4 key extractor to make the process of others decoding the datastream easier.

Would that be of use?
Yes, that would help immensely. Actually, just a reference to instruction that fetches the key from memory would help, then I can trap that in a debugger and fetch it myself :)

AnotherCoreDump
01-18-2005, 09:08 AM
By the way... that might sound like a newbish question but I wonder something about EQ2/EQ1 encryption. Does it encrypt both the EQ Data HEADER and the EQ Data Payload or just the EQ Data Payload?
AFAIK they did encrypt both, header and payload.
The flaw the SEQ Team used for the attack was a known plaintext attack, because before they fixed it you could determine with 99% of the size of the packet what packet type (and with that what header, and with that what kind of plaintext to expect) you had, and thus you were able to get a really fast bruteforce routine to work on some special packets with long known plaintext knowledge. The best thing about that was, that there was a really common packet which did give a really good target for such an attack.

The fix was, that SOE did compress all packets before encrypting them, and thus rendering this kind of attack useless, cause
a) you were not able to tell what packet you had just from looking at the size
b) you were not able to know the plaintext any more cause it was packed

If only 1% of the inbound packets are compressed, you could use the above emntioned technique for a known plaintext attack, but I'm no crypto expert (just did read some books) and don't know who to program such a thing.

AnotherCoreDump
01-18-2005, 09:25 AM
In February 2003 they changed the encryption back to a more ore less simple XOR that made the keysniffer unnecessary. I have no clue why they stepped back, but i assume that there were too much problems with the performance of many pcs and the harder encryption.
AFAIK this is only one reason:
If you look at the history of tools like SEQ and derivates, only a small community of freaks were able to use them (you did need to have a linux box, know how to compile/make something, alot of stuff ...) as long as the DEQ team did not publish the base of the libEQ.a.
And they didn't publish it, because they had some sort of agreement (I think this resulted from some talk with some lead engineers there with some of the seq people because of "copyrighted code" inside the libEQ.a) with SOE.

When SOE release Velious, the SEQ team did delay the release of a new Version of SEQ that did decode the "new" format (changed structures) for some weeks if not a month, again in agreement with SOE.

When SOE did release POP, they got warned from the SEQ developers, that this might lead to a windows version of SEQ because with an active keysniffer, there was no reason any more to have a linux box - why run a program on the linux box, when you have to run a program on the EQ machine anyway and were detectable by that?
So after they did fix the problem, the first versions of MySEQ and other tools popped up.

I thing the reason they switched back was both, resignation (because they now had a really large community that was able to use spy programs with nice windows installers) and performance problems (the encryption and compression did use more cpu power than they had on some machines).
Anyway, why encrypt something when there are alot of programs out there that you can't fool any more? :)

Well, I guess they won't disable encryption for EQII now, because
a) there is no program out there that can break it at the moment
b) most computers have enough cpu power to have it enabled.

Just my opinion about history, btw. Was not really involved. :)

Freakyuno
01-18-2005, 10:23 AM
Sounds like your hinting at a conspiracy theory type of thing between the SOE and the SEQ dev's. This never was the case, although one of the lead dev's of SEQ got hiried by SOE at one point.

The case was an unspoken understanding on the side of SOE, that was controlled by the simple fact that the dev's of SEQ had an almost "moralistic" approach to SEQ. If you werent smart / mature / dedicated / whatever you wanna call it, to get SEQ running, then your exactaly the type of person that couldnt use it responsibly, so you got no help. SOE thought along the same lines evidently, responsibly being the key factor. Of course I'm just guessing here, but to assume that there was SOE copywrited code inside the libEQ.a sounds really silly to me. That would assume that SOE and SEQ were in some sort of long time partnership, sharing code and actually furthering the development of SEQ, instead of just being content that it exists with low numbers.

What this says for EQII, /shrug. Someone will figure it out and break the encryption, SOE will fix it again, and we'll start the cycle over again.

/end Hijack