What I've been up to... (aka ShowEQ 5)

[Zaphod]

Lately I've been working on some major changes to ShowEQ. These changes are very pervasive and aim to both clean up a lot of the code and add new/improved features. When I am done (or at least decide to declare victory) we will have ShowEQ 5.0. I have decided to do this development in a seperate branch within CVS to insulate the regular users from these changes until they stabilize. Once certain milestones are achieved and certain features stabilize I will be releasing beta versions.

Some of the changes related to this development have already been included in (4.3.17, 4.3.17.1, and 4.3.18).

Here are other changes that have already been completed or are nearing completion, but won't be released in 4.3.x:

• Added zone safepoint support to ZoneMgr
• Some dependency cleanups.
• Fix to support people running EQ under winex.
• Switch to using QRegExp for filter pattern matching.
• Major cleanup/revamp of Filter handling (stage 1), filters are now stored in their own sub-directories, and now uses an XML format.
• Created a perl script filter2xml.pl which can be used to convert existing filter conf files to the new XML format.
• More major cleanup of packet handling (stage 2). Migrated opcode,implicitlen and dispatch handling to an XML based system (see worldopcodes.xml and zoneopcodes.xml).
• Modified logging of decoded world/zone packets to include which packet in the opcode DB they matched, when the entry was updated, and information about the type that is supposed to be in the payload.
• Fixed the spawn logger to log the adjusted x/y/z values for zone/new spawns..
• Slight improvement in map loading speeds.
• Properly support seperation of user data from distributed program data, including support for a .showeq directory or the like
• Migrated SEQWindow to be based off of QDockWindow so that all classes derived from it could take advantage of Qt's native dockable window support.

Here is what I currently see as the TODO list for 5.0 (this list will change with items being added and removed):[list][*]Packet handling:[list=1][*]Add support for an EQPacketSession concept that will support one SEQ process to monitor N EQ clients.[*]Organized packet capture display with the ability to filter in/out different
packets. [*]More general packet cleanup.[/list=1][*]Better support for running with SetUID.[*]Move all non-GUI related stuff out of EQInterface[*]Move terminal UI to its own class[*]Move eqstr_{}.txt handling to its own class[*]Improved filtering system[*]Map:

1. Support multiple user configurable icons (for spawns, pc's, items, etc...)
2. Support display of zone safepoint (from zone packet)
3. Support more map editing capabilities.
4. Support simple map optimization
5. Support rescaling the Z for a whole map to allow migration of old/borked maps.[/list=1]
6. Move maps, filters, spawnpoints, and logs to their own subdirectories.
7. New message storage, filtering, and UI.
8. Support for LoY guild information capture/display.
9. Redo the base UI so that it is loaded from a QT .ui file to make it easier for the users to reconfigure/add stuff.
10. Seperate UI from data storage/handling in Combat Window and Experience Window
11. Create new and improved item db support.
12. Improved group/guild/raid support.

If people have specific feature requests I am willing to entertain them (or at least be entertained by them). Also, if people wish to volunteer to contribute to this effort it would be greatly appreciated.

Going forward my 4.3.x work will be limited to opcode and structure changes to keep up with EQLive as well as the occasional bug fix.

Thanks and Enjoy,
Zaphod (dohpaZ)

[S_B_R]

Have I mentioned, it's good to have you back Zaphod?

It's corny I know, but how about a DPS graph in the Combat window? Hit Damage on the Vertical and Time across the horizontal. Mob DPS could be displayed similarly, or even on the same graph in a different color....

Anyway, Thanks for all your hard work Z.

[guice]

Very nice!

Might I make one suggestion? Migrate the Map parsing to use the same format as SOE's. This will assist in cross compatability with in game maps and SEQ.

Didn't see it in your to do list, so thought I'd suggest it.

[Tor K'tal]

Originally posted by guice
Very nice!

Might I make one suggestion? Migrate the Map parsing to use the same format as SOE's. This will assist in cross compatability with in game maps and SEQ.

Didn't see it in your to do list, so thought I'd suggest it.

ACK... how about just double support based on file extention. I know it isn't really the linux way but it would be an easy way to determine which parsing system to use. The reason I say this is because the SEQ map file is technicly superior in file size and other aspects.

How you might ask? The simple answer is the way the line points are stored. SEQ uses x1,y1,[z1,]x2,y2,[z2,]x3,y3[,z3] while SOE uses x1,y1[z1,],x2,y2,[z2]x2,y2,[z2],x3,y3[,z3]. As you can see that will quickly add up to be a much larger files in the long run. Larger files take more time to parse. Also ingame editing of the map in EQ does some weird stuff where it actually can start putting only x1,y1,[z1,]x2,y2[,z2] on a line then putting the next on a new line so it doesn't actually string them all together, meaning you have a line diffinition character, a color set, a point count too. Currently both file formats support the same way to define a color by using an RGB system while SEQ also has the ability to understand pre defined color names such as green and blue and grey and about 12 others. SEQ RGB is formated #RRGGBB in hex and SOE closes their lines with 0 to 255 R, G, B.


So Zaph, got to say what great work you have done so far and I much appreciate it. Support of both file formats wouldn't be too tough based on file extention, or even prompting the user to select SEQ or SOE format if the extention wasn't a .map or a .txt file when they open it manually. The system itself would try and use a .map file if it existed then try for the .txt and if neither existed, no map would be loaded apon zoning.


I think it is an awesome idea to move the opcodes to an XML style system. I just wish I knew more about XML in order to help you with that. I was wondering if you could put the commonly refered to structure in a similar XML sytle format. I am not 100% clear on what this structure is, my believe is it's actual formating of the EQ packets themselves but I could be wrong. I just know that it occationally changes to.

I have a question on the XML for the opcodes, with this system will you be able to edit the opcode on the fly refresh the parse of the XML file and then start interpruting the next amount of data with the fresh changes? I don't expect it to be retro active against data you have already miss interpruted.

So again, great job and keep up the awesome effort.

~ TK

[Zaphod]

Originally posted by Tor K'tal
ACK... how about just double support based on file extention. I know it isn't really the linux way but it would be an easy way to determine which parsing system to use. The reason I say this is because the SEQ map file is technicly superior in file size and other aspects.

I agree. I find the SOE map format to be wasteful both in terms of disk space and processing time. It looks like the work of a very junior level software engineer.

So Zaph, got to say what great work you have done so far and I much appreciate it.

Glad to hear it.

Support of both file formats wouldn't be too tough based on file extention, or even prompting the user to select SEQ or SOE format if the extention wasn't a .map or a .txt file when they open it manually. The system itself would try and use a .map file if it existed then try for the .txt and if neither existed, no map would be loaded apon zoning.

I've been thinking about doing that. It would be fairly trivial to support reading/writing both formats. It's a lower priority item, but could be accomplished in little time.

I think it is an awesome idea to move the opcodes to an XML style system. I just wish I knew more about XML in order to help you with that. I was wondering if you could put the commonly refered to structure in a similar XML sytle format. I am not 100% clear on what this structure is, my believe is it's actual formating of the EQ packets themselves but I could be wrong. I just know that it occationally changes to.

The structures in everquest.h do correspond to the format of the packets themselves. I have been thinking of moving the structures to an XML based system as well, the main concern I have with doing that is most, if not all, of the techniques you would use to do that would be slower then just mapping it to a C style structure. The biggest advantages we'd get are the ease of programatic editing, possibility of runtime correction, minimizing the need for recompiles, and the possibility of supporting big-endian platforms (ala Solaris Sparc, IRIX, and MacOS X).

I have a question on the XML for the opcodes, with this system will you be able to edit the opcode on the fly refresh the parse of the XML file and then start interpruting the next amount of data with the fresh changes? I don't expect it to be retro active against data you have already miss interpruted.

I might support a reparse of the file, but I'm also thinking about the possibility of providing a UI, possibly as part of a packet analysis UI, that allows you to edit the opcodes within ShowEQ while it was running. At which point the change would take effect immediately and then you'd be able to just save your changes to the XML file. This could make opcode hunting much simpler.

So again, great job and keep up the awesome effort.

~ TK

[uRit1u2CBBA=]

How about this:

It would only affect the first time that an SOE map is read per zone, so the overall overhead vs. using only the current SEQ maps are minimal.

I zone into a new zone. Call it ZZZ since it would be the same anywhere.

I look for file named ZZZ.map. If exists, cool, use it. done.

If it doens't exist, then look for ZZZ.txt. If it exists, read it, then write it as a .map, and delete the .txt (if you want to save space).

Now the .map file exists for the next time I zone into ZZZ.

And of course if it doesn't exist in any form, no map, just as it is today.

[Cryonic]

You can almost do that now, it's just not automated. See the Tools Menu, click it the next time you have SEQ open.

[datadog]

Wow Zaphod.. this is totally awesome...

As far as feature requests. Im thinking of some changes to the buff window.

1. Possibly have separate windows for buffs and detrimenal spells like dots and mezzes?

2. Would it be possible to filter the buffs listed in the window to the currently played character (either target or caster)?

Not being a programmer of any ilk, i have no way to know if this is feasible, but I thought I'd ask.

Thanks again for all of this hard work. It is greatly appreciated.

[Velium_hound]

Nice list!

One thing that would be "nice" to have is the a hook for event (packet) listeners. Basically a class that I could add my own instances to which would get copies of the events, decide if it was a event type this listener is interested in, and do something with it.

(Like prices in the bazaar)

The point though is that people could drop in their own experimental toys easily given a standard interface. Then if one of particular merit surfaces, integration is relatively simple.

Personally, I think it would be rather simple to code up a little add-on that might track DPS or "complete heal" trending on a raid. Or alert me when that person hails the PP a second time (or third...).

[jesterbam]

A big fat THANK YOU !!!!!! Amazing skills.

[guice]

I agree. I find the SOE map format to be wasteful both in terms of disk space and processing time. It looks like the work of a very junior level software engineer.

I can agree to that. The idea here is that the SOE Maps is in a format we don't control while the SEQ format is under the SEQ dev's control. It would only make sense to convert what the devs have control over to what they don't have control over, if that makes sense.

Even though SOE's is very poorly done, there isn't anything the SEQ devs, that I know of, can do to fix it. So, just go with the flow.

[monklett]

More robust configuration in logging of spawns, times, locs, etc.

This is the sort of data that is necessary to sort out many quests, etc. (I still haven't quite figured out the SRo AC spawn, although I have killed him over a dozen times).

Even a simple switch to just dump all this to disk, without packet information, would be nice.

[Zaphod]

Originally posted by monklett
More robust configuration in logging of spawns, times, locs, etc.

This is the sort of data that is necessary to sort out many quests, etc. (I still haven't quite figured out the SRo AC spawn, although I have killed him over a dozen times).

Even a simple switch to just dump all this to disk, without packet information, would be nice.

Ummm, most of what you just mentioned can be gotten out of the spawnlog (which doesn't contain packet information). So, what are you requesting that is different then the spawn log?

Enjoy,
Zaphod (dohpaZ)

[monklett]

Originally posted by Zaphod what are you requesting that is different then the spawn log?

I apologize; that post sucked. I shouldn't have flashed it out before bed. What I actually had in mind was log file support for Xylobot and other 3rd party apps.

You see, Xylobot can only accept input from a log file or by reading the colors of specific pixels on the screen (imagine a black screen with only your health/mana bar and chat window). With some programming, like looping a /loc key, you can do rudimentary things like move to a certain location or face a general direction. However, you cannot do things like chase a fleeing mob, or determine the distance between 2 mobs, or even (reliably) face a specific direction.

Of course, Macroquest can do all these things and more. However, it has its own issues in that: (1) its far more detectable, because of the way it reads and writes eqclient information, (2) its scripting language is much less robust (although Xylobot's isn't great either), and (3) it has been available in a functional state far less often than Xylobot over the last couple of years.

However, if ShowEQ could be coaxed into writing only a couple of facts to a disk file every second or so, Xylobot libraries could be easily written to perform nearly all of Macroquest's advanced features, without its issues.

These facts are:

1) your current loc and the direction you are facing in degrees. (I believe EQ's precision is only 2 degrees--at least that true for the client display--I am not sure how it is stored internally).

2) the loc and direction of your current target.

3) (optionally) the loc and direction of any mobs within configurable distance X of your loc.

An update would be made when values change, up to once per second.

When I looked into pulling this information out of the spawn file a few weeks back (you may remember my post about the changes in logger.cpp), it became evident that the best way to do this would be to use a perl or python script to parse the spawn log, strip out the relevant information, and then write it to another file. A better way would be through a config option within ShowEQ to output this information separately, which is what I was looking at doing when I made that post.

(Actually, the right way would be to connect directly to Xylobot via TCP, etc., but that has its own issues. Using the custom log file approach is adequate, well supported within Xylobot, and would allow this information to also be used from any other 3rd party app without any additional tweaks or support issues from the ShowEQ team).

Anyway, I have written some fairly complex Xylobot scripts (including one that will grind AAs without human intervention), and am quite familiar with its quirks and limitations. If this sounds like a feature that you would be interested in, I can provide much more detail on what to output, etc.

[Reason]

Bazaar tracking. /nod/nod =)

Min/max/ave/trends stuff like that. plz.