UESPWiki:Administrator Noticeboard/Archives/Possible Image Fixes

This is an archive of past UESPWiki:Administrator Noticeboard/Archives discussions. Do not edit the contents of this page, except for maintenance such as updating links.

Possible Image Fixes

In terms of our assorted problems with images I have two ideas about what we should do:

1. Change a configuration setting, $wgHashedUploadDirectory, from false to true. I just realized that this one setting could explain a lot of the problems.
2. Upgrade to mediawiki 1.14. The code relating to images has been nearly completely rewritten between MW1.10 and MW1.14, so I'm not sure it's worth investing a lot of time tracking down a problem in obsolete code.

On the other hand, one implication of my recent thoughts is that our problems on db1 are unlikely to be related to the image issues, at least not directly.

With our current setting of $wgHashedUploadDirectory (false), every image on the site is put into the w/images directory (in our case, that's 12,720 files in a single directory). Even worse, every image thumbnail on the site is put into the w/images/thumb directory (assuming 5 thumbnails per image, that comes out to 60,000+ files). That's just way too many files in a directory, so any attempt to access an image file is likely to take extra time -- and those delays will be magnified on content2, where the directory is remotely mounted. Wikipedia and probably every other wiki with thousands of image uses $wgHashedUploadDirectory=true, and that's also the default value of the setting; with "true", the images are reorganized into 256-odd subdirectories, making the number of images per directory much more manageable.

Unfortunately, the reason why we're using $wgHashedUploadDirectory=false is because changing the parameter is not easy. We started out with "false" as the setting, because that was effectively the mediawiki default when UESP was first set up (pre MW-1.4). Furthermore, as stated on the mediawiki manual page "This parameter should not be changed after the first image or file has been uploaded". If the parameter is simply changed (without any other site changes), the wiki will no longer be able to find any of the existing images on the site. There aren't any pre-existing maintenance scripts that can be used to help convert a wiki from one image configuration to the other. I can't even find any information on mediawiki about how to convert (except for one discussion about converting in the other direction).

Nevertheless, it should be possible to convert UESP, and I think it's worth the effort. What I'd like to propose is:

1. Temporarily disable file uploads. I'm guessing this would be necessary for at most one day (possibly only for a few hours). Also, during this period, we shouldn't delete any images.
2. I'll run a script that will copy every image on the site to a new image directory, and place the copied image into the correct hashed subdirectory. This is what will take most of the time; I've already created the script and done test runs of it on my test wiki.
3. At the end of this process, we will have two complete sets of images: one using the hashed=false organization, and one using the hashed=true organization.
4. Edit LocalSettings.php on content1 to simultaneously change $wgHashedUploadDirectory to true and change the images directory to point to the new images directory; check and see whether images are still accessible on content1.
   • If there are any problems on content1, the changes to LocalSettings.php can be immediately undone, reverting images to the original configuration.
5. Assuming everything worked on content1, setup content2 to nfs-mount the new images directory and make the same changes on content2.
6. Re-enable file uploads and we're done. Eventually, delete the original copies of the images to free up the disk space.

The current size of the images directory is 3.6GB, so we have more than enough disk space (115 GB) to duplicate the entire directory.

If this plan seems acceptable, then the question is just when to do it. I'd prefer to not do it over a weekend, however the start of next week is simply not good for me. So the earliest time that I think would be good for this conversion is Wednesday-Thursday (e.g., start at maybe 8pm PDT Wednesday). Any objections to file uploads being turned off temporarily at that point? Or does anyone feel that problems are so bad right now that I should just start the process right away, even if it means there's a risk of triggering new problems when the site is at its busiest?

(Daveh, if you notice this post, there are a couple steps that you need to do: (a) create a new images directory at /home2 with whatever name you think appropriate; make sure it's got the same permissions as wikiimages (b) export that directory (c) mount that directory on content2. These could all be done right away, or else they could be done right before propagating the change to content2.)

As for the other suggestion, namely upgrading to MW1.14, I'll try to spend some time today doing some of the upgrade preparation. --Nephele^Talk 15:00, 6 June 2009 (EDT)

I think I probably speak for most site members when I say "Err... whatever you think best"! As far as I'm concerned, anything that stops the problems we've been having with images is good, and well worth putting up with a brief outage for uploading images.

I assume the upload page can actually be disabled and that we won't have people uploading images that disappear into the ether? –Rpeh^{•T•C•E•} 18:51, 6 June 2009 (EDT)

Yes, the upload page would be disabled. I would set $wgEnableUploads to false for the duration of the conversion, which removes the "Upload File" link from the sidebar, displays an error message if someone finds another way to pull up the Upload page, and makes the wiki refuse to accept any file uploads. All existing images on the site remain completely accessible, however. --Nephele^Talk 19:10, 6 June 2009 (EDT)

Possibly worrying too much here, but... my reply wasn't meant to sound dismissive - it's just that I don't know enough about MW/Linux to do much more than defer to your judgment. As for the upload page - I assumed that's what would happen but thought I'd better check. All sounds good to me. –Rpeh^{•T•C•E•} 19:42, 6 June 2009 (EDT)

I've added /home2/wikiimages2 to content1 and shared it to content2 as requested. Disabled file uploads for a few days shouldn't be much of a problem. Preferably include a noteor something on the upload page saying its temporarily disabled and will be working in a few days. I would wait until after the weekend before starting anything. -- Daveh 20:07, 6 June 2009 (EDT)

Oops, I just noticed one small problem: the permission on /home2/wikiimages2 needs to be changed to 775 (so apache group can write to the directory). At the moment, I'm unable to write any of the new files to the directory. --Nephele^Talk 23:53, 11 June 2009 (EDT)

Given that I couldn't write the images to the correct location, I basically did a dry run tonight then re-enabled file uploads. The entire copy takes less than half an hour, so once the wikiimages2 directory is fixed, it should take about an hour to do everything properly. --Nephele^Talk 01:40, 12 June 2009 (EDT)

Er, should I hold off on image deletions for now? I've noticed the process is pretty sluggish and I don't want to mess with whatever you're working on. –Eshe^talk 01:45, 12 June 2009 (EDT)

It's probably better to hold off. It's not that you're messing with what I'm doing; it's that until some of these changes are implemented, you're going to keep hitting the problems that are making these changes necessary. Problems such as image alterations locking up content2, and files only getting partially deleted because content2 is locked up (e.g., the pictures of Eris Senim and Aevnir). --Nephele^Talk 01:50, 12 June 2009 (EDT)

Well that would explain some of the strange things I've been running into then! I'll work on it later. Thanks Neph :). –Eshe^talk 01:53, 12 June 2009 (EDT)

Images have been reorganized, and we are now using $wgHashedUploadDirectory=true. There are a few quirky workarounds that I took just so I could get the job done now (otherwise it might have been two weeks before I would be able to do this), which means that there are a few more tweaks that should be done eventually. However, any further tweaks can be done without disrupting the wiki.

For anyone who is curious, we had 12,770 files in the images directory, 2595 in the images/archive directory, and 49,070 thumbnails in the images/thumb directory.

If you notice any problems related to images let me know. I'd say if there aren't any reports of problems within the next day or so, then we might as well try image deletion, too, just to see whether it seems to be working any more smoothly (if not, there's still a wiki upgrade to come). --Nephele^Talk 15:07, 12 June 2009 (EDT)