This topic is for technical discussion of overriding scripts -- what's safe and what's not. I suspect that conflicts between a master script and overridden versions of it are responsible for some CTDs.

To make what I'm talking about clearer:
* Suppose that I define a script cobCmpStatusReport in Cobl Main.esm that's supposed to provide a status report on a companion.
* Suppose that "Bob's Better Companions.esp" is built on top of Cobl Main.esm and redefines the script in one way.
* Suppse that "Jim Kicks Bob's Butt Companions.esp" also overrides the same script.

Now suppose that I start playing with just Cobl Main.esm. I then add Bob's which redefines the script. I then add Jim's to load after Bob's. What happens? Or what if I remove both Bob and Jim. What happens then? I'm suspecting CTD (at least under some conditions), but I really don't know.

The main primary suspicion is with script variables -- in particular, with script variables that have been added in the modified esps. I suspect that the safe answer may be: Modifying scripts may never add new variables. It may also be unwise for the original script to remove variables.

It would probably be helpful to understand how mods store and access script variables. I think that's actually fairly well known. See UESP SCPT Mod Record. In particular, for the SCHR record, the VariableCount field is:

It would probably also be helpful to understand how changes to script variables are stored in the save record. That's not so well understood. Essentially, the equivalent to a mod record is a save change record, and for that there's a subfield for the script changes, but we don't know what they are. (See ACHR Script Change for instance.) My expectation is that it will be an an array of variable change records, e.g. "variable 5 now has value 3.2". If this is so, then there is almost certain to be a problem if there's a disagreement among modifying esps over what "variable 5" is. (E.g. float "wickededness" in one and "integer goodDeeds" in the other.)

I can see where more than one override can be problematical, but wouldn't the last in just win out, completely overwriting the first set of changes? Does the whole script get replaced? Or do only subrecords? If the latter, yeah, that would be more of a problem.

How does this compare to the simple issue of changes in scripts with the same savegame though? I add variables all the time in that situation, and my only really identifyable issue so far has been that some of them don't get initialized (because the script has done its doOnce initializations). I can use the console to do that, or retrigger my doOnce flag, and all is well.

Not saying the situations are parallel by any means. I don't know if they are or not. But they seem related. Somehow the savegame isn't choking on the addition of new variables (I JUST did this with my configuration script too.) That's not to say doing this can't have nasty side-effects (as I had with an uninitilized variable leading to a divide by zero problem that drove me nuts), but I haven't seen CTDs I can tie to doing it.

The problem is conflicting tracks of adding variables. This would not be a problem if the updates are only made in a straight line (with is the case for Bethesda). The problem is conflicting lines of adding variables (which is the case for modders and particularly for COBL).

So if I do this:
v01: int alpha int beta
v02: ADD float gamma
v03: ADD int delta
I don't think that's a problem.

The problem I suspect that suppose that Cobl Main does the above, but Bob overrides starting after cobl version 01. And bob does
bob01: ADD refr b1, int b2
bob02: ADD float b3

Worse Jim overrides starting at version Cobl 01:
jim01: ADD int jim01, float jim03, float jim04

Now, for the savegame, what are the variables?
According to Cobl:
vars = alpha, beta, gamma, delta
According to Bob:
vars = alpha, beta, gamma, b1, b2, b3
According to Jim:
vars = alpha, jim01, jim03, jim04
Moreover, those are different types. So, if I switch from Jim to bob, then the float jim04 is now interpreted as a reference b1. Which would be bad.

Again, I'm not sure of that -- it depends on exactly how varables are saved both in mods and in the save game, but I'm pretty sure that something like that would happen.

I understood the concern, but I'm not at all sure how the script revisions interact. If the variables are stored and referenced based on UIDs... it seems to me (based on my limited knowledge) there would only be a problem if there was UID duplication... as DOES happen more often than we would like. That problem is being addressed.

If the variable UID is not stored, but assigned based on sequential position in the script record, yeah, ugly. But is that how it works? Is the storage that static?

Interesting discussion wyre. I would think, in answer to this, like other records the mod loaded last with its version of the script takes priority. And since no actual execution occurs until all mods are loaded, the only script that's going to get run is the last version loaded from a mod, so I don't know that there'd be any concerns of overlapping variables.

Unless, I've misunderstood the problem?

What UIDs? Where is that documented? What's being done? Am I replicating an earlier topic/discussion? If so, let me know and point me to the earlier discussion. (I did look around a bit, but didn't see it.)

Note: Now that you mention UID, I went a double checked the UESP page and saw this: SLSD SubRecord. Which explains uid a little bit, but not how it's generated. (When might different variables have the same UID?) And I'm not certain that it solves the problem -- some run tests would be necessary. Again, a link to and/or summary of earlier discussion (expecially of the solution is desirable.

Martigen: The problem is that load order may change and/or you may remove a mod that you had installer earlier, and maybe replace it with another. E.g., suppose that I have a house using COBL's water wells. Then I use an HTS mod which overrides the script with an HTS version. Then I remove HTS and replace it with a different survival mod which replaces the script with a different version (with different variables). In that case, conflicting variables might be a problem.

Then there's this (from a PM from PhoenixAmon) (in regards to some compilation problems with overriding COBL alchemical sorter scripts). This doesn't make any sense to me:

Well, the formid wouldn't change, and if the internal script variables haven't changed, then why would the uid change?

Hmm... I bet behippo knows the answer to some of this...

Sorry, Wrye, I meant formid. But I'm referring to some internal dynamic means the engine has of referencing data in such a way that it can shuffle the reference when needed (like when an esp overrides an esm.) My understanding is that the formid serves that purpose at some levels. Not sure how far down into the data structures those apply. Maybe there's yet another.

I was referring to the discussions in the Gecko thread a bit farther down and the duplication of formids in different esps that causes problems.

What phoenix is saying points to the formids (or some ID) being what determines what the engine thinks is the variable. And points to at least one issue with dealing with mixing mods. I haven't knowingly run into that situation. I have been running an esm part of the time for some minimods, with esps tied into those, but haven't seen issues with swapping at least the esm the esps depend on (another example is your cobl upgrades, I just dropped them in). But I just may have not hit a combination there that would force me to have to recompile my esps before they will work again. (assuming I'm following phoenix's description) Oh... and I should add I am working on stuff pretty constantly so I may be recompiling and fixing the issue before it creates problems.

I think the only way to KNOW if the scenario you describe can cause a problem is to test it. Stack an esm with two different esps that you force overlaps in and see what results when you mix and match.

Well Ian knows quite a bit more about this than I do. He's been diving into the Script class, but taking a quick look at the header file we have I can give a little information.

There is a Script class which contains all of the information about a script. Ian has also documented a RefVariable structure inside the Script which contains: a TESForm*, a variable Index and some other currently unknown data. There is a RefListEntry structure which is used to build a linked list of RefVariables. Each script contains RefListEntry as the head of list of RefVariables. In addition there seems to be a similar (but not identical) type of structure used by showquest vars.

Ian has hooked up some simple AddVariable and GetVariable functions which retrieve and set variables from the list. Presumably Script itself has similar functions (at least on the Get side), but Ian would be able to say a bit more about it.

There is also information about the number of refs and the length of the compiled script data. He has also decoded a ScriptEventList that keeps track of the various block types.

So, I am not sure how much of this is useful to the discussion - as it doesn't dive into exactly how overridden scripts work.

I haven't done a script overrides to know exactly how it works. I've pinged Ian and asked him to come take a look at the thread, so hopefully he can shed a little more light on the subject.

Ah I see, and so if the mods change and the scripts change, and the player's savegame has different variables stored...



I can say from experience that compiled scripts don't store object names, only the FormID they reference. So if the FormID of an object referenced in a script changes, it needs to be re-compiled to point to the new object. Conversely, and theoretically, if the object name of a given FormID is changed by a subsequent mod loaded, while the CS might baulk that the object name can no longer be found, the script would still run with the correct FormID in game.

Dan: Ahh, okay. Formids and script variables are totally different problems. We've got formids pretty well figured out, and know how they get into scripts (not as variables by as constant references). The duplicate formids problem with Gecko is a technical issue with the way gecko handles merging in objects that seem to be modifying existing formids, but in fact are supposed to be new objects -- i.e., it's not an inherent problem with the engine, but a problem with gecko (inherited from tes4 plugin utility) and the merging process.

It's the variables for scripts that are the problem -- they don't have the protection against change that formids do. Or maybe they do -- maybe Ian can shed some light.

The best I can offer (as usual) is to describe my experience and see what others can make of it. I'll be as clear as I can manage.

I am trying to override some scripts from COBL.esm... my own scripts, so I know what's what in them. The problem part was with 4 quest scripts and the 64 object scripts that set variables in those quests. None of these scripts ought to get a new formID in the overriding esp because I'm just making edits over the esm script.

Quest Script 1: Total rewrite in the override. I have to add a whole mess of new variables to it, but not remove any. Object scripts that set variables in this script simply do nothing when called. The variables are never set. Solution was to open the object scripts and recompile without making any changes. Now the unchanged object scripts exist in the esp as well as the parent esm completely unchanged (on a plaintext level) and now they work.

Quest Script 2: Total rewrite in the override. 1 or 2 variables are removed. None are added. Object scripts that set vars in this script work just fine without changes.

Quest Scripts 3 & 4: No longer needed, except to hold vars for the purpose of staying totally compatible. The script is deleted, a couple of vars are removed. The remaining vars are set properly by the object scripts without a need to recompile the object scripts.

I'd be very interested to hear what others make of this. It took me a long time to figure out that recompiling those unchanged object scripts would do the trick. It seems to me this must have something to do with variables being assigned IDs of some sort in a compiled script. I'm just not sure how that works.

Worth noting: None of the objects that ended up needing their scripts recompiled had any other changes to them before I recompiled the script. The objects, quests and scripts I was overriding were straight from the loaded ESM.

Edit again: Tsk! Wrye, posting at 4 in the morning? You're supposed to wait until February to break your new year's resolutions.

Interesting. It does sound like there's some linkage broken that recompiling reestablishes. The "variables removed" is the apparent commonality with the cases that don't require the explicit recompiling. That probably changes some sort of linked list structure, and perhaps it flag that structure as modified and as a result that IDs in there have changed (maybe they are assigned by position). Simply appending doesn't reassign any, just adds new ones, so no changes to what exists.

That's my theory! (But I'm not sticking to it in the face of better ones.)

I'm still working on decoding how script variable changes are handled in the save file, so I can't be sure yet, but it would be consistent with everything I've seen so far if they didn't handle changes correctly. I've run some tests to see exactly what the runtime behavior is for overriding scripts:

First off, I added a quick command to OBSE to dump out the two variable lists for the current script when called, then made a simple esp that modified the AltarofAkatosh script so it would call the new command when activated. Activating the objects logs this:


Script variables are defined in two tables: one stored on the script object that is shared between all instances of the script, and one stored on what I've called the "event list" above, but is really just per-instance data.

The first two columns in the "script vars" list are unk0 and unk1 in Script::RefVariable. My guess is that they would be used for storage of constant floating point values if the script compiler didn't support inline floats in the script format. Next is a pointer to a TESForm, and in parenthesis the form ID and full name of the form (if valid). Finally is the "varIdx" variable, which is a reference in to the event list table.

The first column in the "event list" table is 'id' in ScriptEventList::Var, the second is 'unk1', and the third and fourth are 'data', represented as first a double and then a UInt64.

To understand what's going on in the tables, it's important to note that the Oblivion script system refers to all variables and form IDs indirectly. The first table ("script vars") is a list of all of the 'ref' variables and form IDs in the script, and the second table ("event list") is a list of all the floating point variables, along with the storage for 'ref' variables. Entries in the first table that represent variables are redirected to the second table by setting the 'varIdx' variable to a non-zero value.

So, with this knowledge and a bit of guessing, we can determine what each entry in the lists is for:

script vars 0: redirects to event list ID 2, representing the "ref target" variable
script vars 1: constant form ID 00000014 (the player), representing the references to "player" in the script
script vars 2: constant form ID 0017720E, I didn't look this up but I'd assume it's the global FameAkatosh
script vars 3: constant form ID 00064C9E (Blessings of Akatosh), brought in by the "cast AltarAkatosh target" line

event list 0: data storage for "short doonce"
event list 1: data storage for "short DayofLastUse"
event list 2: data storage for "ref target"

Next I created another esp that replaced the entire script with:

I added this to the plugin list after my original esp, and got this:

The generated IDs for the script variables started after the IDs generated for the original script. This is good, and means that overriding a script from a master file works correctly. I then saved this game in to a new file. My guess is that the save game now contains data saying "variable 4 contains the data 0x0000000000000014" and etc. Next, I made another plugin swapping the type of the variables, but not the order:


Loading the game showed this list:

This list looks approximately the same as it did from the previous mod, but the variable types are now swapped, so the data is incorrect. Modifying the script so it changes the variable data proves this.

I believe that this is the root of the problem: modifying a script only allocates your new variables after the space allocated for the variables in the master file, and the save file only contains enough information to restore variable data by index, ignoring the type, name of the variable, and which mod it came from. It seems like the variables are allocated IDs in the order they're found in the file, so it may be possible to work around this via only appending variables and never changing existing the order of existing variables.

The crashes to desktop are probably caused when a float is replaced by a ref. Because Oblivion packs the form ID of a ref in to the lower bytes of the double, you'll get a completely random form ID, which probably won't resolve to a form, which results in a NULL TESForm * being passed back through the script command handler, which then probably crashes.

I'm going to try removing variables and a few other things to see what happens, but this post is already very long, so I'll end it here for now.

Wow, fantastic info there, ian!

Looks like we should avoid altering variable declarations in the master.

Sounds like for compatibility reasons it would be better to leave the old variable definition in masters, and add a new variable (and use replace in the script of all instances of the old with the new one), if we need to "change" something relating to an old one. Is that right?

Would this apply equally to changes in esps where we are using old save games? (This might explain some instability I experience in the development process... maybe. Or it could be my PC is trying to die on me ).

I think that we'll have to ruminate on this some more, but one of the approaches for COBL might be to just define a bunch of variables in the cobl version so that overriding scripts don't have to define any new ones. I.e.

scriptName cobWhateverScr

short sAlpha
short sBeta
short sGamma
float f01
float f02
float f03
ref r01
ref r02

etc. (but with better names than that). Obviously, it's got limits but should at least work for simple scripts (e.g. water well activators.)

I'm spending more time on RL these days, so pardon the slow responses... Anyway, I finally had some time to check on the variable situation a little more. Here goes...

First, keep in mind there are three ways you can approach the problem: 1) Mod file script (SCPT) record structure, 2) savegame script change record structure, and 3) runtime array hacking. Ian's view above was through approach number 3 (though I'm not sure how much of that structure was through Oblivion.exe and how much through OBSE code structures).

So, going to approach number one, see: UESP SCPT SLSD/SCVR records. In short:
* Each variable is assigned an index number the first time the compiler encounters it.
* Reordering variable definitions does not change index numbers, nor does changing its type.
* Renaming an variable in effect creates a new variable, so it gets a new index number.
* Index numbers are never reused -- however, when script compilation branches (e.g. different mods modify the same master script), then the different branches will likely define indices in different ways and thus cause a conflict (e.g., the float to ref conflict that Ian suggested before).

Now, going to approach number two, see: UESP Savegam Script Changes. In short,
* All number variables (short, long, float) are stored as (64 bit) doubles. Refs are stored as 32 bit irefs (an indirect referenc to a formid).
* Variables are mapped to mod scripts through the index numbers as defined in the SLSD record (above).
* Variables are scrubbed from the savegame when the master no longer supports their definition (I'm not sure how safe this is, but it may help).

Scrubbing: Supose Master A, then mod B, and mod C, both of which add new, conflicting variable lists to the same script from A. Now suppose a savegame running on mod B, where you want to switch to mod C. To do so: 1) Unload B. 2) Load the savegame and resave (this should scrub the scripts variables back to only those defined by master A). 3) Load C. 4) Load updated savegame and play.

Note: In doing these tests, I did encounter a little weirdness: At one point I had that autospell add error (where a new spell suddenly appears in your inventory while loading the game -- but this may have been a RAM related error as my machine has limited memory and I had a lot of programs running). At another point (after the scrubbing test), I got a CTD on quit -- but not during play. I should mention that these were pretty quick tests.

vs. Ian
All of that looks pretty consistent with what Ian was seeing. Ian's "Script Vars" are really the constant refs of the script -- i.e., the same thing as the SCRO records in the mod file. ("Var" is probably a poor name, since they're actually constants.) His "event list" are the actual script vars, with the first column being the variable index number, and the final numbers being the double and iref mapped to formid of the variable's data

PhoenixAmon's Problems:
I think that probably this was a history problem. I.e., I suspect that at one point, PA deleted the variables from script and then later recompiled with the variables in place. If this happened, then when the variables were redefined upon adding, their indices would not match the old indices. Alternatively perhaps the indices were not simply ordered (e.g., did not appear in the order 1,2,3,4, but maybe like 4,2,3) and OBSE did not handle that correctly. The best thing to do here is probably to check the indices for the variables of the suspect scripts.

COBL Recommendations
I'll have to think about this a little more now that the problem is a little better understood.
* It looks like changes from mod to mod can be dealt with by scrubbing -- a pain, but doable.
* There might be a problem with conflicts between mod an COBL master if COBL ever added more variables to a script -- this might be dealt with by reserving a block of indices for COBL's use. Most likely this will require a special tool to mangle variable indices one way or another.
* Might be a good idea for all cobl variables to be labeled 'cob' (e.g, 'cobShort01'. Might be too much of a pain in the butt though.

Follow up... I coded a litle function up in bish to scan a specified mod file and print out the index/variable list if they were out of order at all. Here's the result (after trimming a couple of longs lists).

Printout is:
scriptId, number of refs (ref vars + ref constants), highest var number
varIndex varName
varIndex varName
etc.
If the variable is a ref (instead of a plain number), I've marked that with [REF] after the varname.

In case it may help, I believe that what I did was significantly change the order that the variables were found in the script. It would never have been compiled with the vars removed because compile would have failed, but I also didn't just paste the new vars in after the old ones. I pasted over the old list of vars with a new and much longer list that was in a different order. Something that I can test out with that same project is whether it would have made a difference if I just added vars to the end of the list, leaving the existing ones alone.

Good to see you back, Wrye.

If you'd like to check the indices on the scripts for your mods, I've just released a version of Bash which has the command available from the command line. See this post.

PA: The reordering of variables shouldn't have made a difference. Perhaps the new bash/bish scriptVars function will help clarify what was happening. The main thing to see is if variables that were defined in the released Cobl Main.esm had their indices shifted for some reason in the code you created.

A couple of other notes: Now that bosh/bish exposes the script file structure, you can do some tweaking with relative ease:
* tweak the lastIndex to determine how the next new variabled will be indexed
* copy the slsd/scvr pairs over from the original cobl main.esm and recompile (only worth doing if the indices have gotten messed up somehow).
* Import text from a text file directly into the code text field.
* Automatically generate scripts and insert them into the esp.
(For these last two, you'd still have to compile the functions in TESC, but they might save some work. I should mention however, that Silk may already do this sort of thing.)

Bump away from forum jaw of death. Unique info here.

just seen this thread. Interesting info and in line with that I figured out myself and what I mentioned here before. As a result of that I made a change to TES4View (will be in the next version) which introduces a classification of "critical conflict". It's currently only used for the Flags field in SLSD and the SCVR record.

Normally if there are only 2 different values TES4View would just mark it as a "override", only if there are 3 different values is it considered a conflict. For fields marked as critical if there are 2 different values (and "no value at all" is not counted) then it's a critical conflict. Because TES4View already lines up groups of SLSD/SCRV records based on the contents of the Index field in the SLSD record this works perfectly.

That means: if a plugin removes a variable defined in it's master it's just an override. If a plugin adds a new variable that's not contained in the master it's an override. If a plugin should defined a variable with the same index but different name or type then it will show up as a critical conflict.

Similar if you have a master and 2 plugins: One or both of the plugins can remove a variable that was defined in the master and it will only show up as override. Variables added in the plugins will show up as override as long as their names and types match (or if one plugin defines more new variables then the other). But if there are any variables for which there is a disagreement on type or name between all the modules which change this script then it will show up as a critical conflict.

I hope this will make it easier to pinpoint all the situations where it's essential to not simply change the load order or remove/add a single module.

Sounds good and useful!

Fortunately it looks as if such conflicts are relatively rare. With all the mods I've installed I've only found a few:

UOP / OOO - ElvenStatueScript
Fran's between the base esm and New items esm - FranLoot1Armor2Script
Qarls's Harvest / DLCHorseArmor / MD Saddle Bags - all the HorsePC*Script's

Wow, that's going to be a very, very useful feature. You rock!

It's included in TES4View 1.2.6 BETA

I saw that! It's looking fantastic!

Bumpity. Crucial info no doc'ed anywhere else AFAIK.

Has anyone been able to determine whether global (variable) reference index changes in a script also suffer from this problem?

What I'm talking about here are the script vars x examples that ianpatt gave, not the event list x examples.

Shouldn't. The SCRO's would be recompiled every time the script was recompiled, thus the script code would always be in sync with the list of SCRO's. The only way you would have a problem is if you externally modified the SCRO list.

Thanks, man!