Hedgehog TDS 5.1.0.9 and Sitecore 8 Update 3

Older versions of TDS have issues with Sitecore 8 Update 3, and they can manifest in a few different ways.

When installing a package that was generated using an older version of TDS, you might encounter an error about the following error...

Exception Could not find file 'C:\Inetpub\wwwroot\GasTest\Website\_DEV\DeployedItems.xml'.

In the installation screen on the Update Installation Wizard it would look like...

Or on the results screen...

Also, when you are using Visual Studios or a TFS build server to generate the package, the build might fail with the very descriptive error...

C:\Program Files (x86)\MSBuild\HedgehogDevelopment\SitecoreProject\v9.0\HedgehogDevelopment.SitecoreProject.targets (165): The package builder failed. Please see the build output log for more details.

Digging into the logs would give you the stack trace...

Building .\bin\TestCM-SC8\..\Package_TestCM-SC8\TDS.Master.scitems.update
Inner Exception Object reference not set to an instance of an object.(System.NullReferenceException):
at Sitecore.Update.Utils.ConfigurationUtils.GetConfiguration(ConfigReader reader)
at Sitecore.Update.Utils.ConfigurationUtils.GetConfigNodes(String xpath, ConfigReader reader)
at Sitecore.Update.Configuration.Factory.ReloadSettings()
at Sitecore.Update.Commands.BaseFileCommand.Serialize(XmlWriter writer, SerializationContext context)
at Sitecore.Update.Commands.SerializationCommandFactory.SerializeCommand(ICommand command, XmlWriter writer, SerializationContext context)
at Sitecore.Update.Installer.CommandToEntryConverter.ConvertFileOperationCommand(BaseFileCommand entry, Stream file)
at Sitecore.Update.Installer.CommandToEntryConverter.Convert(ICommand entry)
at Sitecore.Update.Installer.CommandToEntryConverter.InternalConvert(ICommand entry, IProcessingContext context)
at Sitecore.Install.Framework.BaseConverter`1.Convert(T entry, IProcessingContext context)
at Sitecore.Install.Framework.BaseSource`1.InternalSink.Put(T entry)
at Sitecore.Install.Framework.FilteringSink`1.Put(T entry)
at Sitecore.Update.Installer.CommandSource.InternalPopulate(ISink`1 sink)
at Sitecore.Install.Framework.BaseSource`1.Populate(ISink`1 sink)
at Sitecore.Install.PackageProject.InternalPopulate(ISink`1 sink)
at Sitecore.Install.Framework.BaseSource`1.Populate(ISink`1 sink)
at Sitecore.Install.Utils.EntrySorter.Populate(ISink`1 sink)
at Sitecore.Install.PackageGenerator.GeneratePackage(PackageProject solution, ISink`1 writer)
at Sitecore.Update.Engine.PackageGenerator.GeneratePackage(DiffInfo diff, String licenseFile, String outputPath)
Exception Cannot generate package: Object reference not set to an instance of an object.(System.Exception):
at Sitecore.Update.Engine.PackageGenerator.GeneratePackage(DiffInfo diff, String licenseFile, String outputPath)
at HedgehogDevelopment.SitecoreProject.PackageBuilder.PackageBuilder.g(g ?)
at HedgehogDevelopment.SitecoreProject.PackageBuilder.l.g(String[] ?)
5>C:\Program Files (x86)\MSBuild\HedgehogDevelopment\SitecoreProject\v9.0\HedgehogDevelopment.SitecoreProject.targets(165,5): error : The package builder failed. Please see the build output log for more details.

Fixing this error is as easy as downloading the latest version of TDS. This issue was fixed as of 5.1.0.9. More specifics about this can be found in the following links...

http://www.hhogdev.com/blog/2015/may/sitecore-update-installation-wizard-error-in-sitecore-8-update-3.aspx

http://jockstothecore.com/packaging-with-tds-and-sitecore-8-update-3/

Sitecore 8 update 3, CD Servers and Index sitecore_list_index

I recently completed an upgrade to Sitecore 8 update 3. I ran into some issues after configuring the CD server following the recommended Sitecore documentation...

After configuring the CD server, I was getting numerous errors in the log file about "Index sitecore_list_index was not found".

It turns out that the sitecore_list_index should not actually be deleted from the CD, yet the standard SwitchMasterToWeb.config was doing so.

Support was kind enough to provide me with an updated SwitchMasterToWeb.config, which fixed the issue. I'm not sure if their official one has been changed yet, so I'm making the one they gave me available here for now.

SwitchMasterToWeb.config

Care When Packaging Item Buckets

If you ever have to create a package for items within an Item Bucket, take care to include all the parent folders along the item path.

If you only include the items themselves, Sitecore doesn't remember that they used to be in Item Buckets and the folder structure it creates to store them in will be of the template "System\Node".

Let's say you want to create a package with the following items. The "Courses" folder is your top-level Item Bucket...



The quickest and easiest way would be to go to the "58" folder, and use the "Add with subitems" button.



However, if the Sitecore instance you install the package on does not already have that exact Item Bucket folder structure... it will create the folders using the "System\Node" template.

I've ended up with folder structures that are a mix of Item Buckets and System\Nodes because of this, and according to Sitecore's support this is something to be avoided as it could potentially cause problems.

So to be safe, go through and add each Item Bucket folder in the chain, as well as any items you want to add. It's more tedious, but it will let you avoid having to Sync a bunch of System/Node folders after you've installed your package.

Mass Delete Through Serialization

Item Buckets in Sitecore 7 are great, but I've run into a problem with them. Deleting mass quantities of data through the UI can be slow. I have one bucket that has ~27k records of data underneath it that I want to clean out and re-import. Any time I try to delete this folder and its children, the site times out again and again.

I tried writing a quick method call to run the delete. I added a button that I could click from the Content Editor, which would make an async call to a method that deleted all the children of the folder. This worked with no time-out, but it took a very long time to loop through all the children and delete everything.

Then I was introduced to a trick using the Serialization functionality inherent in Sitecore.

Sitecore has the ability to serialize content items into text files. This is essentially what TDS is doing behind the scenes. Serializing the content items and managing the created files for you. These Serialization options can be found in the Developer ribbon.

You can also Revert Tree on a content item, which reads the serialized text files and builds the content item and its children accordingly. This is where the trick comes in.





For our example we have a content item called "Courses". The children of this item represent the ~27k pieces of Course data that I have imported and want to delete.






 Check that the content item has been serialized. You can check this by going to the path C:\inetpub\wwwroot\CCBC\Data\serialization\master\sitecore\content\Data\CCBC, and looking for the "Courses" folder and "Courses.item" serialized file. If it hasn't been serialized, then go back to the content item and click the "Serialize Item" button in the Developer ribbon.

If the entire tree has been serialized, you can go into the Courses folder and see more folders and files that represent the structure that Item Buckets build when items are added to them.

 

Go into the Courses folder and delete everything in there. You want this folder to be empty.

Now go back to the Courses item in the Content Editor. Click on "Revert Tree" in the Developer tab. You'll get a dialog box that might last a few minutes while it does its thing.

Once it's done, refresh the Courses folder and you'll see that all of its children are gone!

In the end, this Serialization method took maybe 3-4 minutes to delete ~27k records. Doing it programmatically was taking 15-30+ minutes (not to mention code time, deploying, etc.). Deleting through the UI just wasn't happening at all.

There is one small issue with this however. You'll notice that if you do a search on the Item Buckets folder, it will still return a result count. No actual items get returned, but Sitecore still *thinks* it has returned all the items that used to exist.

This will remain until you do a full index rebuild. Just doing a "Re-Index Tree" isn't enough for this one. This is an issue known by Sitecore, the "Re-Index Tree" functionality only deals with new or updated records. It won't clear deleted items from the index.

Refresh Tree, Partial Re-Indexing

I've been doing a lot of work recently with large data loads, and I've been trying to find ways to optimize the time that my site has to spend rebuilding indexes.

It used to be, after each data import I would kick off a full index rebuild programmatically. Because of the large amounts of data and computed fields, each rebuild was taking a long time and led to me using Hot Swappable Indexes.

An optimization I found that helped with this was how to only re-index the new data that I just added. You can call the method IndexCustodian.RefreshTree() on a folder, and only it and its children will be re-indexed.

The code for this...

     var database = Sitecore.Data.Database.GetDatabase("master");
     Item dataFolder = database.GetItem(new ID(Settings.GetSetting("CourseFolderId")));
     SitecoreIndexableItem indexableFolder = new SitecoreIndexableItem(dataFolder);
     Sitecore.ContentSearch.Maintenance.IndexCustodian.RefreshTree(indexableFolder);

This is essentially the same functionality as clicking the Re-Index Tree button in the Developer ribbon.

Doing this reduced the time spent on re-indexing after an import by a large amount.

One side-note, the RefreshTree() method returns a collections of Jobs that you could then monitor their progress if you wanted. You can read more about Jobs here.

Hot Swappable Indexes with Sitecore 7

Recently I had a project that was heavily using somewhat complex computed fields to index a large amount of data. Because of this, rebuilding the entire index was taking a substantial time.

The problem was that our search pages for the site relied on these indexes to function. So anytime a full index was being kicked off, the search would basically be unavailable for the time it took the index to rebuild.

Thankfully, I found out that in Sitecore 7 there is a new feature that lets me avoid this problem. A configuration setting called SwitchOnRebuildLuceneIndex, that essentially rebuilds the index in a second directory. While the index is building, the site runs off the first directory as normal. When it's finished, it swaps over to using the second freshly updated index. No downtime in between.

This was really easy to setup to. Just go to the configuration file Sitecore.ContentSearch.Lucene.Index.Master.config (and don't forget to repeat this for the Web index, and any other instances you have), and replace the following line...

<index id="sitecore_master_index" type="Sitecore.ContentSearch.LuceneProvider.LuceneIndex, Sitecore.ContentSearch.LuceneProvider">

with...

<index id="sitecore_master_index" type="Sitecore.ContentSearch.LuceneProvider.SwitchOnRebuildLuceneIndex, Sitecore.ContentSearch.LuceneProvider">

That's it! Now, rebuild the index twice, and you should notice a new directory in your indexes folder.

I wish this was a default setting, it would have saved me a little headache.

Managing Configuration Files With Multiple Developers

The Problem:
This is something that has come up several times now. A client will have a Sitecore instance. There will be multiple developers on the project. There will be multiple environments (local developer machines, QA, Staging, Production, etc.).

How do you keep all the configuration files organized and clean, so developers aren't overwriting each other and each environment gets the correct files?

The Previous Solution:
Up until now, I've usually went with separate projects to keep all the config files organized. Folders for each developer, each environment and any global files that didn't need to be changed. Then I would use TDS's File Replacement to handle copying the required files over to the Website folder on build.

There are several problems with this though. It's somewhat confusing to setup. You end up having to make the same change to multiple config files. It didn't handle instance-wide config files (ones that had to span multiple projects) very well.

The New Solution:
Enter configuration transformations.

Effectively, this will allow you to have 1 master configuration file for each specific config file and create transformations for any specific information that needs to be changed (ie: connection strings for different developers).

Things you will need for this setup...
 * TDS
 * Visual Studio 2012
 * Slow Cheetah, a Visual Studio Extension

For this example we have a very simple solution...

Here you have 2 projects. Your empty Web Application and a TDS project. Delete the Web.config that was automatically added, and add in the Web.config from your Sitecore install.

Next, setup a configuration for yourself in Visual Studio's Configuration Manager.

Right click the Web.config and select the "Add Config Transform" option. You may get a popup about the project having to be reloaded. Make sure you have any changes saved and click "Yes".

You should now see a new configuration transformation added to your project.

This new transformation file is where you add any environment specific changes. For instance, each developer's "dataFolder" might be slightly different. So in the transformation file (ie: Web.Sarkis.config) you tell it to find and set the attribute for the "dataFolder" variable in the Web.Config.

<?xml version="1.0" encoding="utf-8"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <sitecore database="SqlServer">
    <sc.variable name="dataFolder" value="C:\Inetpub\wwwroot\MyCustomInstall\Data\" xdt:Transform="SetAttributes" xdt:Locator="Match(name)"/>
  </sitecore>

  <system.web>
    <pages configSource="App_Config\Custom\WebControls.config" xdt:Transform="Replace" />   
  </system.web>
</configuration>

You can then preview what your configuration file will look like after all the transformations have been applied to it. Just right click on the transformation file you want to preview (ie: Web.Sarkis.config), and select the "Preview Transform" menu option. As you can see, the "dataFolder" string has been changed to use my personal value.

You can now setup in your Web project any and all config files you might use in your Sitecore project. Just mimic the folder structure you would find in your Sitecore's Website folder (ie: \App_Config\Include), create Configuration Manager settings for any environment you might install to and create transformation files for each of the config files.

Now we use a combination of TDS and Slow Cheetah to get these configuration files to your Sitecore's Website folder on build.

Install TDS and add a TDS project to your solution. In the Configuration Manager, create a custom configuration for the new TDS project. Once this has been done, right click on the TDS project and select "Properties".

In the General tab, you will set the "Source Web Project" to be your Web Application project.

In the Build tab you will set the Web Url of your Sitecore install, as well as your local installation location of the Website folder.

Save these changes.

Next, go to Visual Studio's extensions and updates. From the Tools menu, click the "Extensions and Updates" option. In the "Online" tab search for "slow cheetah", download and install the extension. Normally configuration transformations only take place when you publish a site. This extension allows it to take place on a build.

Once TDS and Slow Cheetah have been setup, just build your solution and the Web.config (and other config files) with the proper transformations applied to it should be copied to your Sitecore Website folder.

If you use TDS' Update Package capabilities, the files in the update package that gets generated will also have the transformations applied to them.

This makes deployment to different environments much easier. Just create a Configuration Manager setting for that environment, create the transformation file that only contains the differences, build the solution and then deploy the update package through Sitecore's Update Installation Wizard.