New Method to obtain opcodes with little effort

[fester]

HOWTO find opcodes by examining eqgame.exe:

It seems the opcodes change frequently and it is a rather involved hunt to find the new opcodes each patch, I decided we should investigate the possibility of determining opcodes just as we do with the decode.cpp file.

Examining the code, I did not find any switch blocks of the proper size. I then decided to look for jmp to data located within the data segment:

egrep 'jmp[ \t]*ds:off' = 147 hits (excluding switch jumps = 79)

Add the last cmp before the jump:

egrep '[ \t]cmp|jmp[ \t]*ds:off' disasm.txt | grep -v switch.jump | awk '/cmp/ {c = $0} /jmp/ {print c; print $0}'

Looking thru the 158 lines produces only one likely target:
.text:00421821 cmp eax, 2BAh
.text:00421834 jmp ds:off_424F34[eax*4]

This is the only spot with a cmp/ja combo used to skip a jump of this format. I examined several archives of previous code
listings. They all only have one code clip of this type:

.text:00421AF1 cmp eax, 2AFh
.text:00421B07 jmp ds:off_42539C[eax*4]

and

.text:004910E2 cmp eax, 2A3h
.text:004910FB jmp ds:off_494AEA[eax*4]

OK, so we found the dispatcher for opcodes:

.text:00421818 loc_421818: ; CODE XREF: sub_4217B0+1A^Xj
.text:00421818 add eax, 0FFFFFFE9h
.text:0042181B xor ebx, ebx
.text:0042181D push esi
.text:0042181E mov esi, [ebp+10h]
.text:00421821 cmp eax, 2BAh
.text:00421826 push edi
.text:00421827 ja loc_423B3B
.text:0042182D movzx eax, ds:byte_4251D0[eax]
.text:00421834 jmp ds:off_424F34[eax*4]

First subtract 0x17 (23) from the wire opcode (0x0017 becomes 0x0000), skip above 0x02BA, fetch from a hard code table located at byte_4251D0 with this value as the index. What you end up with is the translated opcode (this value remains static in every copy of eqgame.exe I have on hand.)

0xA6 likely means "invalid opcode" inside this table (and is used for padding as well as an indication of how many real opcodes are present.)

To make it such that all opcodes are mapped within a few hours of any patch, one must map EVERY opcode now. Then translate them into the native opcodes. Once in native format, when eqgame.exe is updated, reverse the mapping to obtain the "wire opcodes". The native functions do not appear to change (excluding issues like new opcodes added; structures changed; opcodes removed entirely.)

For the curious I will include a dump of valid opcodes below.
"wire opcode" (wire opcode - 23) = native opcode:

opcode 17 (0) = 0
opcode 1f (8) = 1
opcode 21 (a) = 2
opcode 23 (c) = 3
opcode 27 (10) = 4
opcode 3b (24) = 5
opcode 3d (26) = 6
opcode 42 (2b) = 7
opcode 43 (2c) = 8
opcode 54 (3d) = 9
opcode 60 (49) = a
opcode 62 (4b) = b
opcode 65 (4e) = c
opcode 67 (50) = d
opcode 70 (59) = e
opcode 72 (5b) = f
opcode 74 (5d) = 10
opcode 75 (5e) = 11
opcode 76 (5f) = 12
opcode 7f (68) = 13
opcode 80 (69) = 14
opcode 93 (7c) = 15
opcode 95 (7e) = 16
opcode 96 (7f) = 17
opcode 97 (80) = 18
opcode 9e (87) = 19
opcode 9f (88) = 1a
opcode a0 (89) = 1b
opcode a1 (8a) = 1c
opcode a2 (8b) = 1d
opcode a6 (8f) = 1e
opcode a7 (90) = 1f
opcode ab (94) = 20
opcode ad (96) = 21
opcode b1 (9a) = 22
opcode b2 (9b) = 23
opcode b7 (a0) = 24
opcode bc (a5) = 25
opcode be (a7) = 26
opcode bf (a8) = 27
opcode d1 (ba) = 28
opcode d2 (bb) = 29
opcode d5 (be) = 2a
opcode da (c3) = 2b
opcode e0 (c9) = 2c
opcode e4 (cd) = 2d
opcode e6 (cf) = 2e
opcode ec (d5) = 2f
opcode ee (d7) = 30
opcode fb (e4) = 31
opcode 102 (eb) = 32
opcode 103 (ec) = 33
opcode 104 (ed) = 34
opcode 10e (f7) = 35
opcode 10f (f8) = 36
opcode 110 (f9) = 37
opcode 11e (107) = 38
opcode 11f (108) = 39
opcode 121 (10a) = 3a
opcode 124 (10d) = 3b
opcode 126 (10f) = 3c
opcode 128 (111) = 3d
opcode 12d (116) = 3e
opcode 12e (117) = 3f
opcode 133 (11c) = 40
opcode 135 (11e) = 41
opcode 13b (124) = 42
opcode 142 (12b) = 43
opcode 146 (12f) = 44
opcode 148 (131) = 45
opcode 149 (132) = 46
opcode 14a (133) = 47
opcode 14e (137) = 48
opcode 150 (139) = 49
opcode 152 (13b) = 4a
opcode 154 (13d) = 4b
opcode 155 (13e) = 4c
opcode 161 (14a) = 4d
opcode 164 (14d) = 4e
opcode 165 (14e) = 4f
opcode 167 (150) = 50
opcode 169 (152) = 51
opcode 174 (15d) = 52
opcode 17c (165) = 53
opcode 17f (168) = 54
opcode 180 (169) = 55
opcode 185 (16e) = 56
opcode 187 (170) = 57
opcode 18f (178) = 58
opcode 190 (179) = 59
opcode 192 (17b) = 5a
opcode 19a (183) = 5b
opcode 19c (185) = 5c
opcode 19f (188) = 5d
opcode 1a2 (18b) = 5e
opcode 1a6 (18f) = 5f
opcode 1a9 (192) = 60
opcode 1ac (195) = 61
opcode 1af (198) = 62
opcode 1b4 (19d) = 63
opcode 1b8 (1a1) = 64
opcode 1b9 (1a2) = 65
opcode 1be (1a7) = 66
opcode 1c0 (1a9) = 67
opcode 1c5 (1ae) = 68
opcode 1c6 (1af) = 69
opcode 1c8 (1b1) = 6a
opcode 1cc (1b5) = 6b
opcode 1cd (1b6) = 6c
opcode 1d4 (1bd) = 6d
opcode 1da (1c3) = 6e
opcode 1e0 (1c9) = 6f
opcode 1e8 (1d1) = 70
opcode 1e9 (1d2) = 71
opcode 1ea (1d3) = 72
opcode 1ec (1d5) = 73
opcode 1ed (1d6) = 73
opcode 1ee (1d7) = 74
opcode 1ef (1d8) = 75
opcode 1f3 (1dc) = 76
opcode 1f6 (1df) = 77
opcode 1fa (1e3) = 78
opcode 1fb (1e4) = 79
opcode 1fc (1e5) = 7a
opcode 1fd (1e6) = 7b
opcode 200 (1e9) = 7c
opcode 201 (1ea) = 7d
opcode 203 (1ec) = 7e
opcode 204 (1ed) = 7f
opcode 206 (1ef) = 80
opcode 207 (1f0) = 81
opcode 209 (1f2) = 82
opcode 20a (1f3) = 83
opcode 224 (20d) = 84
opcode 22b (214) = 85
opcode 230 (219) = 86
opcode 231 (21a) = 87
opcode 232 (21b) = 88
opcode 233 (21c) = 89
opcode 24d (236) = 8a
opcode 24e (237) = 8b
opcode 252 (23b) = 8c
opcode 25d (246) = 8d
opcode 25e (247) = 8e
opcode 25f (248) = 50
opcode 262 (24b) = 8f
opcode 264 (24d) = 90
opcode 26f (258) = 91
opcode 272 (25b) = 92
opcode 275 (25e) = 14
opcode 279 (262) = 93
opcode 27b (264) = 94
opcode 27e (267) = 95
opcode 27f (268) = 96
opcode 280 (269) = 97
opcode 282 (26b) = 98
opcode 283 (26c) = 99
opcode 284 (26d) = 9a
opcode 286 (26f) = 9b
opcode 289 (272) = 9c
opcode 28b (274) = 9d
opcode 296 (27f) = 9e
opcode 2af (298) = 9f
opcode 2bb (2a4) = 9a
opcode 2bf (2a8) = a0
opcode 2c1 (2aa) = a1
opcode 2c5 (2ae) = a2
opcode 2c8 (2b1) = a3
opcode 2d0 (2b9) = a4
opcode 2d1 (2ba) = a5

edit: Why I don't have smilies turned off by default, death to all smilies.

[LordCrush]

Wow

I dont say that i unterstand it ( lack of assembler knowledge ) but it looks great

/smile

[Ratt]

That's all the opcodes? Seems like there should be more, but that could just be because they are such a pain in the ass

Very good work. Now if we can map them all to in game action/responses, we're in good shape.

[serberus]

correct me if i'm wrong, but shouldn't this be a relatively "simple" thing to do now?

By that I mean, shouldn't a huge number of people be able to use a simple memory sniffer with the appropriate address for an opcode and then do numerous things in EQ to see if they are able to determine what each opcode does, simply by trial and error?

I originally though Zaphod had posted a list of memory addresses to be checked, but I re-read it and slightly altered my post.

Is locating the addresses of the opcodes in the first place the difficult task while my first point is the easy bit?

I'm afraid I don't understand ASM so I can't be hugely constructive, but i'm very keen to assist by offering my time and man power on simple tasks and such.

Thanks

Serberus

[BlueAdept]

Great I got some of it, but lost you after the line below:

"HOWTO find opcodes by examining eqgame.exe:"

JK, I do understand some of it, but it is still over my head. I havent used my assembly knowledge since the 80's when I used to crack disk based copy protection.

[monklett]

It seems the opcodes change frequently and it is a rather involved hunt to find the new opcodes each patch, I decided we should investigate the possibility of determining opcodes just as we do with the decode.cpp file.

Its a cool idea and clearly the way this should work, but if it were me, I would wait on this until a month or so after LDoN is released. In my experience, every expansion modifies the EQ datastream substantially, as do the assorted post-expansion fixes that come out over the following month or so. Remember when Luclin came out?

After that, EQ usually settles down to a lengthy period of reletive stability.

[Zaphod]

fester:
Which dissasembler did you use to generate disasm.txt? and what flags did you use? I like the looks of the generated disassembly better then the results I get with objdump. I can of course update your egrep/awk commands to it's formats, but I prefer the output from whatever you're using.

Dispatcher for obcodes generated by 'objdump -d -M intel'

  421818:       83 c0 e9                add    eax,0xffffffe9
  42181b:       33 db                   xor    ebx,ebx
  42181d:       56                      push   esi
  42181e:       8b 75 10                mov    esi,DWORD PTR [ebp+16]
  421821:       3d ba 02 00 00          cmp    eax,0x2ba
  421826:       57                      push   edi
  421827:       0f 87 0e 23 00 00       ja     0x423b3b
  42182d:       0f b6 80 d0 51 42 00    movzx  eax,BYTE PTR [eax+4346320]
  421834:       ff 24 85 34 4f 42 00    jmp    DWORD PTR [eax+4345652]

You are right, once someone finds this section it should be fairly straight forward to do the conversion (structure and opcode addition/deletion not withstanding). Good work fester.

Thanks and Enjoy,
Zaphod (dohpaZ)

[Zaphod]

I compared the theoretical wire opcodes resulting from your algorithm to those known to be valid for the 08/26/03 EQ patch release and I found that something is amiss. Numerous valid and verified opcodes do not appear in your list. Perhaps a6 doesn't indicate an invalid opcode and some other processing occurs in the function pointed to by a6 or perhaps some other area pre-cooks the opcode. I'm not sure what's missing.

Verified wire opcodes that are missing from your list include (but are not limited to):

#define PlayerPosCode               0x0022 //
#define MobUpdateCode               0x0039 //
#define StartCastCode               0x00bb //
#define ActionCode                  0x00db //
#define EmoteTextCode               0x00eb //
#define MakeDropCode                0x00f3 //
#define NewCorpseCode               0x00fe //
#define SpawnUpdateCode             0x013a //
#define ClientTargetCode            0x016c //
#define FormattedMessageCode        0x01dd //
#define sWhoAllOutputCode           0x0212 //

Any ideas?

Enjoy,
Zaphod (dohpaZ)

[fester]

Originally posted by Zaphod
fester:
Which dissasembler did you use to generate disasm.txt? and what flags did you use?

IDA Pro and basically defaults on the flags. Windows app, but worth it.

[fester]

Originally posted by Zaphod
I compared the theoretical wire opcodes resulting from your algorithm to those known to be valid for the 08/26/03 EQ patch release and I found that something is amiss.

I examined three opcodes and they all checked out (by hand.) I did not examine the handler for those opcodes. I did look at the calling routine (which is the code in charge of handling FLAG_COMBINED packets and it appeared to recursively call for each sub packet/opcode.)

Possibly I should have spent more than an hour on this project prior to posting. Just looked at it again, and it doesn't appear (in a 15 min look) to alter the opcode data from the wire.

It may be as simple as the table I extracted from eqgame.exe was corrupt?

I will be out of pocket until possibly Thurs, at which point I will take a closer look.

Here is the calling routine:
.text:004217B0 sub_4217B0 proc near ; CODE XREF: sub_4217B0+46^Yp
.text:004217B0 ; sub_49A392+12A^Yp
.text:004217B0
.text:004217B0 var_ABC = dword ptr -0ABCh
.text:004217B0 var_AB8 = dword ptr -0AB8h
.text:004217B0 var_AB4 = dword ptr -0AB4h
.text:004217B0 var_AB0 = dword ptr -0AB0h
.text:004217B0 var_AAC = dword ptr -0AACh
.text:004217B0 var_AA8 = dword ptr -0AA8h
.text:004217B0 var_AA4 = dword ptr -0AA4h
.text:004217B0 var_AA0 = dword ptr -0AA0h
.text:004217B0
.text:004217B0 mov eax, offset loc_595CF3
.text:004217B5 call sub_58087C
.text:004217BA sub esp, 0A90h
.text:004217C0 mov eax, [ebp+0Ch]
.text:004217C3 push ebx
.text:004217C4 test ah, 20h
.text:004217C7 mov [ebp-10h], ecx
.text:004217CA jz short loc_421818
.text:004217CC push dword ptr [ebp+14h]
.text:004217CF push dword ptr [ebp+10h]
.text:004217D2 push eax
.text:004217D3 lea eax, [ebp-40h]
.text:004217D6 push eax
.text:004217D7 call sub_4A3767
.text:004217DC add esp, 10h
.text:004217DF test al, al
.text:004217E1 jz short loc_421811
.text:004217E3 xor ebx, ebx
.text:004217E5
.text:004217E5 loc_4217E5: ; CODE XREF: sub_4217B0+5F^Yj
.text:004217E5 push dword ptr [ebp-38h]
.text:004217E8 mov ecx, [ebp-10h]
.text:004217EB movzx eax, word ptr [ebp-40h]
.text:004217EF push dword ptr [ebp-3Ch]
.text:004217F2 push eax
.text:004217F3 push dword ptr [ebp+8]
.text:004217F6 call sub_4217B0
.text:004217FB cmp dword_76FE88, ebx
.text:00421801 jz short loc_421811
.text:00421803 lea eax, [ebp-40h]
.text:00421806 push eax
.text:00421807 call sub_4A38C0
.text:0042180C test al, al
.text:0042180E pop ecx
.text:0042180F jnz short loc_4217E5
.text:00421811
.text:00421811 loc_421811: ; CODE XREF: sub_4217B0+31^Xj
.text:00421811 ; sub_4217B0+51^Xj
.text:00421811 mov al, 1
.text:00421813 jmp loc_424F25

[Zaphod]

Originally posted by fester
I examined three opcodes and they all checked out (by hand.) I did not examine the handler for those opcodes. I did look at the calling routine (which is the code in charge of handling FLAG_COMBINED packets and it appeared to recursively call for each sub packet/opcode.)

Possibly I should have spent more than an hour on this project prior to posting. Just looked at it again, and it doesn't appear (in a 15 min look) to alter the opcode data from the wire.

It may be as simple as the table I extracted from eqgame.exe was corrupt?

I will be out of pocket until possibly Thurs, at which point I will take a closer look.

I ran a simple program to do the extraction myself and came up with the same table as you did using your algorithm. So I don't think your eqgame.exe was corrupt (or else mines corrupt the same exact way). I just think we're either missing or misinterpreting something.

I'm also going to be out through Thursday.

Enjoy,
Zaphod (dohpaZ)

[fester]

ConsiderCode 0x0155

opcode 155 (13e) = 4c

0x4c * 4 + 00424F34 = 00425064 which results in
jumping to loc_422AA9.

I traced the code at loc_422AA9 and it is certainly consider code.

.text:00422AA9 loc_422AA9: ; CODE XREF: sub_4217B0+84^Xj
...
.text:00422ABF call sub_493CB7

.text:00493CB7 sub_493CB7 proc near ; CODE XREF: sub_4217B0+130F^Xp
...
.text:00493CCA push 2FD0h
...
.text:00493CD5 call sub_4B77DE

sub_4B77DE picks strings from eqstr_*.txt to display and 0x2FD0 is "Consider who?"

Also picked up another easy to spot opcode:
sub 0x3E = "you gained party exp" (0x12d wire opcode at loc_423146), can someone get some party exp and verify this code is sent?

Will look at it more tonight. I am still not sure why some of the know opcodes going forward are incorrect. I could not locate some of those opcodes to go in reverse as easily as these two.

[fester]

Also, ChannelMessage 0x001f (wire opcode) seems ok.

Also 0xA6 points to the code that every opcode jmp's to when it has completed its work. So anything matching 0xA6 essentially becomes a nop.

This table is used for opcodes which do not contain the FLAG_COMBINED (0x2000) flag:

test ah, 20h
jz loc

I assumed the code below that point split up the opcodes, and did not otherwise alter the opcode. I will try to get some time tonight to verify this theory.

Anyway this brings up the question:

Are all the opcodes which appear to be broken coming down the wire with FLAG_COMBINED set?

[Zaphod]

Originally posted by fester
Also, ChannelMessage 0x001f (wire opcode) seems ok.

Also 0xA6 points to the code that every opcode jmp's to when it has completed its work. So anything matching 0xA6 essentially becomes a nop.

This table is used for opcodes which do not contain the FLAG_COMBINED (0x2000) flag:

test ah, 20h
jz loc

I assumed the code below that point split up the opcodes, and did not otherwise alter the opcode. I will try to get some time tonight to verify this theory.

Anyway this brings up the question:

Are all the opcodes which appear to be broken coming down the wire with FLAG_COMBINED set?

Ok, just had a look at some of the opcodes that don't match and how they come across the wire. It would appear that as a set they are independent of whether or not FLAG_COMBINED is set.

The following opcodes come down both with and without FLAG_COMBINED set (come down both individually and as part of FLAG_COMBINED packets):

• PlayerPosCode
• MobUpdateCode
• EmoteTextCode
• MakeDropCode
• NewCorpseCode
• SpawnUpdateCode

I have never seen the following opcodes be part of a FLAG_COMBINED set (always come down by themselves):

• StartCastCode
• ClientTargetCode

The following opcodes seems to always be part of a FLAG_COMBINED set:

• ActionCode
• FormattedMessageCode

Enjoy,
Zaphod (dohpaZ)

[Zaphod]

Ok, upon examination of the code section beginning at 004217B0 and ending with 00421834 I've found that it handles packets both with and without FLAG_COMBINED set.

Here's my annotated analysis of the code as I currently understand it:

; 04217b0 - begin of function that handles dispatching packets
.text:004217B0 sub_4217B0 proc near ; CODE XREF: sub_4217B0+46^Yp 
.text:004217B0 ; sub_49A392+12A^Yp
.text:004217B0
.text:004217B0 var_ABC = dword ptr -0ABCh
.text:004217B0 var_AB8 = dword ptr -0AB8h
.text:004217B0 var_AB4 = dword ptr -0AB4h
.text:004217B0 var_AB0 = dword ptr -0AB0h
.text:004217B0 var_AAC = dword ptr -0AACh
.text:004217B0 var_AA8 = dword ptr -0AA8h
.text:004217B0 var_AA4 = dword ptr -0AA4h
.text:004217B0 var_AA0 = dword ptr -0AA0h
.text:004217B0
.text:004217B0 mov eax, offset loc_595CF3                 ; begin packet handling
.text:004217B5 call sub_58087C
.text:004217BA sub esp, 0A90h
.text:004217C0 mov eax, [ebp+0Ch]                         ; retrieve opcode from stack
.text:004217C3 push ebx
.text:004217C4 test ah, 20h                               ; check if opcode is FLAG_COMBINED
.text:004217C7 mov [ebp-10h], ecx
.text:004217CA jz short loc_421818                        ; if not FLAG_COMBINED then jump to opcode dispatch
.text:004217CC push dword ptr [ebp+14h]
.text:004217CF push dword ptr [ebp+10h]
.text:004217D2 push eax
.text:004217D3 lea eax, [ebp-40h]
.text:004217D6 push eax
.text:004217D7 call sub_4A3767                            ; call to help unpack FLAG_COMBINED packets (uses implicit length table)
.text:004217DC add esp, 10h                               ; pop stack
.text:004217DF test al, al                                ; test for success / packet count?
.text:004217E1 jz short loc_421811                        ; goto finished packet handling
.text:004217E3 xor ebx, ebx
.text:004217E5
.text:004217E5 loc_4217E5: ; CODE XREF: sub_4217B0+5F^Yj  ; loop over sub-packets in FLAG_COMBINED packets
.text:004217E5 push dword ptr [ebp-38h]
.text:004217E8 mov ecx, [ebp-10h]
.text:004217EB movzx eax, word ptr [ebp-40h]
.text:004217EF push dword ptr [ebp-3Ch]
.text:004217F2 push eax
.text:004217F3 push dword ptr [ebp+8]
.text:004217F6 call sub_4217B0                            ; call ourselves (packet handling)
.text:004217FB cmp dword_76FE88, ebx
.text:00421801 jz short loc_421811                        ; goto finished packet handling
.text:00421803 lea eax, [ebp-40h]
.text:00421806 push eax
.text:00421807 call sub_4A38C0                            ; anorther call to help unpack FLAG_COMBINED packets (uses implicit length table)
.text:0042180C test al, al                                ; test for success / packet count?
.text:0042180E pop ecx
.text:0042180F jnz short loc_4217E5                       ; yes, loop around and handle next sub-packet
.text:00421811
.text:00421811 loc_421811: ; CODE XREF: sub_4217B0+31^Xj  ; finished handling packet
.text:00421811 ; sub_4217B0+51^Xj
.text:00421811 mov al, 1
.text:00421813 jmp loc_424F25                             ; jump to end of packet dispatch that will ret 0x10
.text:00421818 loc_421818: ; CODE XREF: sub_4217B0+1A^Xj  ; Dispatch opcode function
.text:00421818 add eax, 0FFFFFFE9h                        ; subtract 27 from opcode
.text:0042181B xor ebx, ebx                               ; clear ebx
.text:0042181D push esi
.text:0042181E mov esi, [ebp+10h]
.text:00421821 cmp eax, 2BAh                              ; check that opcode is below 0x2ba
.text:00421826 push edi
.text:00421827 ja loc_423B3B                              ; jump to completed handling (function A6's address)
.text:0042182D movzx eax, ds:byte_4251D0[eax]             ; get function code from byte in table
.text:00421834 jmp ds:off_424F34[eax*4]                   ; call packet function (base + (function code * 4))

I'm not sure why certain opcodes don't appear to be processed by this code. Tommorrow I may try brushing up on my WinDbg and try to set a breakpoint to trigger on one of these opcodes.

fester, how'd your analysis go?

Enjoy,
Zaphod (dohpaZ)