User:Daveh/ESO Update

The UESPWiki – Your source for The Elder Scrolls since 1995
Jump to: navigation, search

This page details the procedure of updating the UESP data after a major ESO update/release.

General Prep and Notes[edit]

  • Run ESO patcher to get latest version.
  • (Optional) Edit content3:/home/uesp/esolog/esolog.index and increment/note the log index. Optional but useful if you need to find a specific entry in the new logs.
  • (Optional) Backup/delete the uespLog_backupData.txt file created by uespLogMonitor.
  • PTS/Live updates are slightly different. Most steps are for both updates while some steps are for only one or the other.
  • Note that the VERSION field used for scripts/databases is just the update number itself or the update number with pts appended to it. For example:
  • Live: 17
  • PTS: 17pts

Export Game Data[edit]

  • Edit createNewExport.sh and update the top 4 lines with the relevant update numbers
  • Example for Live Update (using update 17):
    VERSION="17"            # Game update number
    ISPTS=""                # 1 for PTS, blank for Live
    LASTVERSION="16"        # The last game update
    LASTPTSVERSION="17pts"  # The last PTS game update
  • Example for PTS Updates (using update 17pts);
    VERSION="17pts"         # Game update number
    ISPTS="1"               # 1 for PTS, blank for Live
    LASTVERSION="16"        # The last game update
    LASTPTSVERSION="16pts"  # The last PTS game update
  • Run createNewExport.sh (typically takes 4-6 hours to run, creating 1000k files, using 200GB).
  • Fix/address any errors that occur during the script.

Update Log Parser[edit]

  • Edit the parseLog.php file.
  • Update the START_MINEITEM_TIMESTAMP and real-time timestamps.
  • Update the MINEITEM_TABLESUFFIX and SKILLS_TABLESUFFIX constants with the current version.
  • (Optional) Update the $IGNORE_LOGENTRY_BEFORE_TIMESTAMP1 field with the relevant timestamp. Typically this is done in conjunction with clearing the logEntry table every 6 months or so to prevent that table from becoming too large and slowing down parsing.

Update esofiles.uesp.net[edit]

  • Create new directory files1:/shared/uesp/esofiles/update-XX/
  • Upload all ZIP files created in the export process.

Update esolog/esoCommon.php[edit]

  • Add API version to $APIVERSION_TO_GAMEUPDATE and $APIVERSION_TO_GAMEVERSION arrays.
  • Update the GetEsoItemTableSuffix() function with the latest update numbers.
  • Live: Update the GetEsoItemTableSuffix() function to return the correct current update number (usually done after item mining is completed).
  • Live: Update the GetEsoUpdateVersion() function to return the current update number (usually done after item mining is completed).
  • Update set indexes. Copy content from the Sets.php file generated by the data extraction script and copy into esoCommon.php replacing the existing $ESO_SET_INDEXES variable.
  • Optional: Edit and run updateSetIds.php if you need to update the internal game set ID in the setSummary table after you run createSetSummary.php or in another table.

Skills and Champion Points[edit]

  • Only dump data on a character with no CPs/skills purchased and wearing no items that could affect the mined data. Also any add-ons that modify skill icons (ex: LuiExtended) should be disabled.
  • /uespdump skills abilities VERSION
  • /uespdump skills types VERSION
  • /uespdump cp VERSION
  • Upload saved variable files.
  • Run log parser.
  • Only for Update 28 and Earlier: Edit createCpFits.php with the new table $TABLE_SUFFIX and run.
  • Edit fixSkillsSlotted.php with the new table $TABLE_SUFFIX and run.
  • Edit fixDestructionSkills.php with the new table $TABLE_SUFFIX and run.
  • Edit fixPvpArtifactSkills.php with the new table $TABLE_SUFFIX and run.
  • Edit createSkillTree.php with the new table $TABLE_SUFFIX and run.
  • Update and run php71 generateSkillIcons.php to create skill icons for tooltips API (after icons are uploaded).
  • Update raw skill data and tooltips:
  • The first step can be done as soon as the files are extracted from eso.mnf.
  • Run TestSkillFormat to create the esoRawSkillData.php file from the skill data in eso000.mnf (modify the one input parameter to be VERSION).
  • Copy to the esolog/esoRawSkillDataVERSION.php file.
  • Update table suffix at top of parseRawSkillData.php.
  • The next step has to be done after the skill coefficients have been updated and fixed (see below section for details).
  • Run parseRawSkillData.php. Pay attention and fix any warnings/errors displayed (I usually only fix player skills afterwards from the database since there tends to be a lot of minor errors).
  • Run the following commands in ESO to update the flag data:
  • Reset all CP/skills and remove equipment.
  • Buy CPs needed for the various commands below. Remember to unequip prior CPs at each step.
  • Run: /usc savedesc (this may need to be done at multiple points)
  • Equip the +DOT Damage CP and run: /usc checkdesc dot
  • Equip the +DIRECT Damage CP and run: /usc checkdesc direct
  • Equip the +ST Damage CP and run: /usc checkdesc stdmg
  • Equip the +AOE Damage CP and run: /usc checkdesc aoedmg
  • Equip the +DS Duration CP (Shield Master), equip and run: /usc checkdesc ds
  • Equip the +DS CP (Bastion) and run: /usc checkdesc ds
  • Buy the +HEALING DONE CP and run: /usc checkdesc heal (keep this purchased during the next healing steps)
  • Run: /usc savedesc
  • Equip the +HOT Healing CP and run: /usc checkdesc hot
  • Equip the +AOE Healing CP and run: /usc checkdesc aoeheal
  • Equip the +ST Healing CP and run: /usc checkdesc stheal
  • Not Needed Since Update 32: Buy the DK World in Ruin passive and run: /usc checkdesc flameaoe
  • Equip the Elf Bane set and run while in combat: /usc checkdesc elfbane
  • Equip the Diamond’s Victory set and run while in combat (trigger with a ranged spell): /usc checkdesc melee
  • Upload log and parse.
  • After Items Mined: Run: findSetBonusSkills.php to link scaling sets to their matching skill.
  • (Live Only) Move old tables (minedSkills, skillTree, skillTooltips, minedSkillLines, cpSkills, cpSkillDescriptions, cpDisciplines) to tableOLDVERSION and new tables to tables with no version suffix, for example:
     RENAME TABLE minedSkills TO minedSkills16; RENAME TABLE minedSkills17 TO minedSkills;
Can now do this using the script (VERSION is the current update version):
      php renameTables.php skill VERSION

Update Skill Coefficients[edit]

  • Only dump data on a character with no CPs/skills purchased and wearing no items.
  • When equipping items only use items/sets with "simple" modifiers. For example, a set that increases spell damage of Fire based skills will affect skills differently and doesn't update the character's weapon/spell damage stat and may result in inaccurate coefficients. Prefer items/sets that just modify base stats and apply to everything. Avoid any +healing, +damage shield, and any cost modifiers (including traits).
  • Add all skills that have a description containing a number:
    /usc addall
  • Use an add-on like Alpha-Gear to save item setups. On PTS this doesn't always work since characters are copied/deleted frequently.
  • Equip a complete setup of Light Armor and run the command:
   /usc save
Wait for the save command to finish before running the next command.
  • Unequip 2 random pieces of armor, save, and repeat until naked. This gives around 6 points of data for each Light/Medium/Heavy setup which is a good balance between too few points (less accuracy) and too many points (more accuracy but time and space consuming).
  • Repeat with a complete setup of Medium and Heavy armor.
  • Run the command:
    /usc calc
  • Reload the UI. Copy/backup the (large) saved variable file.
  • Run the command:
   /usc reset
  • Reload the UI and upload the (smaller) saved variable file to the parser. Parse the new data.
  • Check for coefficients that didn't compute correctly. This regularly occurs due to skills being updated/changed. Easiest way to directly query the database using a query like:
    SELECT id, name, rank FROM minedSkillsXX WHERE isPlayer=1 AND R1<0.99 AND R1>0;
Repeat the query for R2...R6. Note that R values from 0.90-0.99 might be considered "valid" or "good" in some cases.
  • Fix the skill coefficient data found in the uespLog.SKILLCOEF_SPECIALTYPES variable within uespSkillCoef.lua as needed.
  • Restore the large saved variable backup file in order to recalculate coefficients (or simply redo all steps from scratch).
  • Recalculate the skill coefficients by running the commands:
    /uespreset log
    /uespreset temp
    /usc calc
Repeat the backup/upload/parse procedure until all bad coefficients have been fixed.

Mine Item Data[edit]

  • Make sure you do this on a max level character (CP160) with no CP/skills purchased and not wearing any ability altering equipment.

PTS[edit]

  • Only the min/max item data are exported in PTS updates to minimize the time and storage required.
  • Initialize the item summary data (used for detecting item data corruption):
        /umi createsummary
        /umi checksummary
If there are any issues detected in the last command you may need to restart the client and re-create the summary data.
  • Run the following in-game commands:
        /umi table pts
        /umi autostart 1
  • After each automatic /reloadui copy the uespLog.lua saved variables to a new file for later manual upload (or do it automatically with uespLogMonitor).
  • Repeat the mining for potion data.
        /ud uespLog.MinePotionData()
        /umi table safe
        /umi potion on
        /umi autostart 0
  • Item mining for PTS takes about 2 hours and there should be around 16 saved variable files.
  • Upload files and parse.
  • Edit createMinedItemSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
  • Edit createSetSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
  • Edit updateSetIds.php, update the $TABLE_SUFFIX variable with the current version and run (do this after updating $ESO_SET_INDEXES).

Live[edit]

  • Initialize the item summary data (used for detecting item data corruption):
        /umi createsummary
        /umi checksummary
If there are any issues detected in the last command you may need to restart the client and re-create the summary data.
  • Run the following in-game commands (make sure level/type/item type/potion options are turned off):
        /umi table safe
        /umi autostart 1
  • Note that the entire process takes around 1 week of full time mining to complete.
  • Repeat the mining for potion data.
        /ud uespLog.MinePotionData()
        /umi table safe
        /umi potion on
        /umi autostart 0
  • Save the valid item data by the command (replace XX with the current version):
       /umi idcheck XX
  • Edit createMinedItemSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
  • Edit createSetSummary.php, update the $TABLE_SUFFIX variable with the current version and run.
  • Edit updateSetIds.php, update the $TABLE_SUFFIX variable with the current version and run (do this after updating $ESO_SET_INDEXES).
  • Delete fixitems.lua.
  • Update version in findMissingItems.php and run.
  • Copy output found in fixitems.lua to uespLog.lua and run /ud uespLog.StartMineTest(1, false) in-game to mine those missing items. Upload log, parse and re-run item summary scripts.
  • Repeat for findBadItemSets.php, findBadItemSummaries.php, findBadItemSetDescs.php, and findBadItemNames.php until no more bad items are found.
  • Run: findSetBonusSkills.php to link scaling sets to their matching skill.
  • Move old tables (minedItem, minedItemSummary, setSummary) to tableOLDVERSION and new tables to tables with no version suffix, for example:
     RENAME TABLE minedItem TO minedItemXX; RENAME TABLE minedItemYY TO minedItem;
Can now do this using the script (version is the current update version):
      php renameTables.php item VERSION
You may need to restart Apache on content3.

Update Log Viewer (PTS Only)[edit]

  • Only needed for PTS updates.
  • Edit viewLog.php and change the ENABLE_PTS_VERSION constant to the current update version.

Update Maps[edit]

  • Create a 400% for all combined maps (this can be done any number of ways, the PhotoShop version is given below):
  • (Do Once) Create an action to resize 400% using "Preserve Details 2.0" resampling with 100% reduce noise.
  • Run an Image Processor using the previous action on all images in goodimages-VERSION\CombinedMaps and output to goodimages-VERSION\LargeMaps.
  • Typically takes 2-3 hours to generate all maps.
  • Usually only have to do new/updated maps found in CombinedMapsNew or CombinedMapsNewPts.
  • Create the zoom levels 11 and 12:
  • Run the Python script: CreateTilesFromLargeMap.py VERSION e:\esoexport\
  • Copy tiles to NewMaps: ./export/CopyNewMapTiles1112.sh VERSION
  • Upload any files in the NewMaps folder to content1/2/3:/home/uesp/www/esomap/ and files1:/shared/uesp/maps/esomap/.
      rsync -avz ./ dave@content3.uesp.net:/home/uesp/www/esomap/
      chown dave:uespadmin -R /home/uesp/www/esomap/
      chmod g+rwx,o-w -R /home/uesp/www/esomap/
      rsync -avz /home/uesp/www/esomap/ dave@content1.uesp.net:/home/uesp/www/esomap/
      rsync -avz /home/uesp/www/esomap/ dave@content2.uesp.net:/home/uesp/www/esomap/
      rsync -avz /home/uesp/www/esomap/ dave@files1.uesp.net:/shared/uesp/maps/esomap/
  • Edit maps_new.txt that contains a list of all new maps to add.
  • Edit the second column to be the displayed name.
  • Edit the last column to be the ID of the parent world. For new worlds change this to be -2 so the database can be manually updated later. Otherwise enter the worldId as found from the ESO map if applicable.
  • On db1 find the last user world ID by the SQL command: select max(id) from uesp_gamemap.world;
  • Edit createNewMaps.sh with the first free world ID (+1 from the previous step) and update version and run.
  • Upload the newmaps.sql to db1.
  • Backup the current uesp_gamemap database:
      mysqldump --opt -u user -p uesp_gamemap > gamemap.sql
  • Run the SQL file:
      mysql -u user -p uesp_gamemap < newmaps.sql
  • Run php makeZoomLinks.php to update Leaflet symlinks for the new map.
  • Test the map to ensure new maps have been successfully added.

Update Game Data[edit]

Achievements[edit]

  • This is usually only done for Live updates.
  • /uespdump achievements VERSION
  • Upload log and run log parser.
  • Copy old esoAchievementData.php to esoAchievementDataXX.php
  • Run createAchievementData.php

Collectibles[edit]

  • This is usually only done for Live updates but can be done for PTS updates in order to fully update the uespLog runebox data.
  • /uespminecollect
  • /ud uespLog.MineCollectibleTree()
  • Run createRuneBoxData.php (after mined items are updated) and update uespLog.RUNEBOX_COLLECTIBLE_IDS array in uespLog.lua with any new runebox/collectible pairs.
  • Upload log and run log parser.
  • (Live Only) Copy old esoCollectibleData.php to esoCollectibleDataXX.php. Copy data out of tempData in the uespLog.lua saved variable file and re-format/paste into esoCollectibleData.php.

Books[edit]

  • This is usually only done for Live updates.
  • Perform this on a character that has unlocked Eidetic Memory/Shalidor's Library.
  • Run /ud uespLog.MineBookTree() in-game.
  • Run /ud uespLog.MineBooks() in-game (upload/parse log data).
  • Copy data from the tempData section of the saved variable file, re-format, and paste into esoBookCollectionData.php.
  • Upload books.sql to db1 and run.

Quests[edit]

  • This is usually only done for Live updates.
  • Update the esoQuestData.php file from the Quests/Quest.php file created by the export script.
  • NOTE: This step is no longer required due to a change in quest parsing.
  • Upload quests.sql to db1 and run.

Recipes[edit]

  • This is usually only done for Live updates.
  • Perform this on a PTS template to ensure all recipes are recorded correctly.
  • Run /ud uespLog.MineRecipeDataStart() from in-game.
  • Copy data from the tempData section of the saved variable file, re-format, and paste into esoRecipeData.php.

Sets[edit]

  • Update the setInfo table with new set types and locations.
  • Edit and run generateSetImages.php to create image tooltips if needed.

Styles / Motifs[edit]

  • This only has to be done if new motifs/styles were added or existing styles changed.
  • Use /ud GetHighestItemStyleId() in-game to get the last style ID.
  • Use /ud GetItemStyleName(#) in-game to get the style material.
  • Use /ud GetItemStyleMaterialLink(#) in-game to get the style material and find the icon name.
  • Use /uespstyle all, /uespstyle allid in-game to list all styles.
  • Edit uespLog.lua and update the uespLog.CRAFTSTYLENAME_TO_ITEMSTYLE and uespLog.CRAFTSTYLENAME_TO_MOTIFID arrays.
  • Edit uespLogTradeData.lua and update the uespLog.ALT_STYLE_ICON_DATA and uespLog.STYLE_ICON_DATA arrays.
  • Edit esolog:esoCommon.php and update the $ESO_ITEMSTYLE_TEXTS array.
  • Edit esochardata:viewCharData.class.php and update the $MOTIFS_FOR_MASTERWRITS array.

Traits[edit]

  • This only has to be done when new traits are added.
  • Edit esolog:esoCommon.php and update the $ESO_ITEMTRAIT##_FULLTEXTS, $ESO_ITEMTRAIT_DESCRIPTIONS, $ESO_ITEMTRANSMUTETRAIT_IDS, and $ESO_ITEMTRAIT##_TEXTS arrays.
  • Edit esolog:resources/esoitemlink.js and update the ESO_ITEM_TRAITS array.
  • Edit esolog:resources/esoItemSearchPopup.js and update the UESP.ESO_TRAIT_TYPES array (and select lists if needed).

Antiquities[edit]

  • Usually only done for live updates.
  • Run /ud uespLog.MineAntiquities() in-game and upload/parse the log file.

Other Data[edit]

  • esolog:esoCommon.php $ESO_ITEMTYPE_TEXTS
  • esolog:esoCommon.php $ESO_ITEMSPECIALTYPE_TEXTS and $ESO_ITEMSPECIALTYPE_RAW_TEXTS
  • Some constants can be checked by running the /ud uespLog.MineApiConstants() command and then extracting the text from the tempData section of the saved variables file.
  • esolog:esoCommon.php $ESO_SET_INDEXES (from Sets.php created by the createNewExport.sh script)

Update Icons[edit]

  • Upload art to content3:
    rsync -avz ./esomnf-XX/esoui/art/ dave@content3.uesp.net:/home/uesp/www/eso/gameicons/esoui/art/
    rsync -avz ./gamemnf-XX/esoui/art/ dave@content3.uesp.net:/home/uesp/www/eso/gameicons/esoui/art/
  • Update file permissions:
    sudo chown dave:uespadmin -R /home/uesp/www/eso/gameicons/
    sudo chmod a+rx,o-w -R /home/uesp/www/eso/gameicons/
  • Copy from content3 to files1:
    rsync -avz --checksum /home/uesp/www/eso/gameicons/ dave@files1.uesp.net:/shared/uesp/esogameicons/

Update API[edit]

  • This is only done for Live updates.
  • Ensure your file system is setup with case-insensitive filenames. (see below for Windows).
  • Modify EsoParseData.py and update the INPUT_GLOBAL_FILENAME, INPUT_LUA_PATH, and OUTPUT_PATH variables.
  • Run EsoParseData.py from a CygWin shell window.
  • If script aborts in error fix the folder security permissions in Windows and retry.
  • Edit makeApiDiff.sh to the current and previous API versions and run.
  • Upload the new API directory to content2:/home/uesp/www/eso/data/.
  • Copy api.html to the root index.html.
  • Replace the symbolic link current to the new API directory.
    ln -sfn 100025 current

Update Sales Items[edit]

  • Only done for Live updates.
  • Once items have been mined and are live run the script:
    php fixNewSalesItems.php
to update the data of any new items that have appeared in sales data since the update.

TODO[edit]

Notes[edit]

  • Modify Windows to use case insensitive filenames in Cygwin shell.
  • Install the Linux Subsystem in Windows.
  • Run the command:
    fsutil.exe file setCaseSensitiveInfo "full path of folder" enable
  • This doesn't change any existing sub-folders but should enable it on any new sub-folders created after the command is run.
  • Old Method (No Longer Works) As of ~2018 the below method seems to no longer work.
  • Set Windows registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\kernel\obcaseinsensitive=dword:00000000.
  • Reboot after modifying registry key.
  • Use Cygwin with posix=1 set for all relevant drives in /etc/fstab.
  • Test case insensitivity before running script.
  • (Optional) Set Windows registry key obcaseinsensitive back 1 after finished.