What is the methodology used to determine the packet structure?

[joojooga]

I would guess it all starts with a packet capture using something like Ethereal or Sniffer. But once you have the data, how would you go about determining where frames and headers begin and end? And what is the methodology used to decrypt the data in those if they are encrypted?

[Poncho]

I'm in the process of learning this myself as this same exact post seems to trickle out everytime SEQ is not working for some amount of time...I think it was Ratt that hit the nail on the head in one of his replies..(sorry too lazy to find the actual thread)...but it went something like this:

Most of the responses in a nutshell just say to goto an empty zone and start doing shit. You'll notice changes and tendancies in your data. This should, in theory help you find what you are looking for.

Unfortunately I'm not good at this (yet) and cant help more than that. I also realize that there is really no way to put into words how to teach someone how to do this....Just a matter of "time behind the line" so to speak.

Poncho

[fester]

Capture every packet.

Isolate the parts that change and the parts that stay the same.

If you determine that it is compressed or encrypted, look at the data a short while to see if it is clear what protocol encoded it. If it is not quickly clear, open a disassembler and look at the source code.

Once you know exactly how to decode the data, look at the unencoded data. Do the same action over and over again until you isolate struct values and their meanings. If your lazy, look at your disassembled output and obtain from it the size of all the elements of a struct (especially useful if there are very large structs like item and char profile structs.)

Really very easy, but can take time and considerable patience.

[joojooga]

Excellent. Thx both replies. I think I'll take a shot at it. I didn't even consider a most excellent point. Piddle around in game doing actions you *know* what the data will be.

YOu'll know your Loc, what spell you cast, what zone youre in, level etc...

[Alfred]

Originally posted by fester
Capture every packet.

If you determine that it is compressed or encrypted, look at the data a short while to see if it is clear what protocol encoded it. If it is not quickly clear, open a disassembler and look at the source code.

Once you know exactly how to decode the data, look at the unencoded data.

I've been a C and then java programmer for the 'professional' world for about 13 years and I think the above is a lot harder than it sounds. One would have to have intimate knowledge of the disassembler, the assembler language, and what diferent protocol/compressions look like. Not to mention that back when I was using a disassembler... not all disassemblers are created equal. Maybe that has changed...

[baelang]

SOE is using pretty standard libraries for things like compression.

[fester]

I made the assumption that the user had a rudimentary knowledge of disassembling, Intel assembly, and many "common" compression and encryption methods.

[SilentVoice]

Been looking at some of the code lately. Does anybody have a stub program that will decompress/unencrypt a specific and specified section of the captured data? If not, I'll look into writing my own. I just hate to waste time starting from scratch if someone already has it written.

[Amadeus]

You all realize that if someone wrote a decent HOWTO on this topic, we could greatly increase the number of available devs on SEQ.

Personally, I feel pretty comfortable manipluating the code, and doing my own custom changes (adding features as well) to SEQ...however, I'm utterly clueless on how you guys determine packet structure and opcodes, in all honesty.

[fester]

You can not write a HOWTO on this task (at least I would not know where to start.)

It is like using a handgun. Some people are natural and some people have to work rather hard to be a good shot.

All this is based off hunches. You look at the data and you just know what it looks like. If you do not know, then you have to spend a bunch of hours looking at the source.

[Amadeus]

Yes...but writing a bit about the exact software, decryption software, etc... and perhaps just a simple example of one structure found (screenshots) ...would be excellent.

[joojooga]

Hey fester,

I have no knowledge of any of that stuff. I do know that one can achieve a working knowledge of just about anything by studying and understanding the overall concepts and methodologies of a particular field.

That having been said, I've been forging ahead with my quest at about 15 minutes a day with C++ as my starting point . I purchased a book called Ivor Horton's Beginning C++ by Wrox. He really drills down on the boring side of C++ but it's the fundamentals that I'd like to understand firmly. Another book that I work alongside that simultaneously is called Teach yourself visual C++ in 21 days. Radically different book. The fundamentals of the one do however "snap into" the second one.

I took a screenshot of a typical ethereal capture of Everquest traffic and put it up here. This is typical of EQ traffic. UDP back and forth. I believe all the information you need is in the DATA portion of the UDP packet. Now that I see the packet's data I ask could myself a million questions..

1. What is it encrypted with?
2. Is it compressed? How do I know if it's compressed?
3. What am I looking at? Was that me conning something or zoning, etc?

I already have a background in networking so I have an advantage in that area. I chose my weak side (programming) for study so that once I break the information out of the packets I can use the programming side to take that information and make something meaningful. (A mob list, a map, levels..etc..) But how does one "break the information out of a packet?"...Perhaps in my study of C++, I will learn that maybe it's easier for me to pull the data out of memory instead of decoding packets and send that on..who knows...

As you can see, for those interested in this pursuit, there's a lot of work to be done to catch up to the expertise of the developers on this board. I agree with Amadaus that there needs to be a thread that relates the general outline of what to do, and the tools used in each step. I also agree with fester that you could never write a full HOWTO on this. But we CAN develop a HOWTO on the process and leave the details to the people learning, because when the student is ready, the teacher will arrive.

As for those that are already there, I am truly impressed. I just hope that the fundamental building blocks I am learning now get me to where you are in this lifetime.

-Joojooga

[high_jeeves]

I think you guys are overcomplicating the issue. You start a capture program, you do things in game.. you look at the packets as you do things.. (sit and start over and over in an empty zone, and watch the packets).. do this with a variety of actions, look for patters, identify those patterns.. As for compression and encryption: disassemble the eqgame.exe for information on these... i use w32dasm to do this, but there are a variety of others that will do the job just fine.. like fester said, there isnt really a "right way", we just look at the disassembly, look at the packet stream, and spend time figuring it out..

Also, read the ShowEQ source.. although it doesnt work right now, it will give you an idea of some basic structures (since they have changed, they arent 100% accurate), and some of the basic compression information..

--Jeeves

[Amadeus]

Hmm...thanks Jeeves for a great posting! Personally, I know how to find offsets in eqgame.exe from my workings with MQ. And, I suppose if I had some "roadmaps" from before, I could probably find offsets for ShowEQ using the same method (ie, looking for familiar packets, etc..).

However, no one really ever explained how one determines a structure with MQ...only how to find offsets.

[high_jeeves]

You just look at it, and figure it out.. there is no magic formula here... look at all of the spawn structures, look for similarities and differences.. figure out why a byte changes, and for what reasons... thats it.. its just pure common sense, there is no methodology..

--Jeeves