Devops with Sitecore Tools & process

Part-1 | Part-2

Our platform is based on our beloved e-CMS and Experience platform Sitecore. 

For any deployment at high-level there are three key artifacts code, config & data. In usual plain ASP.NET project data means T-SQL scripts or database backups or if ORM is there, then entities/modals. But in case of Sitecore usually data exchanged between systems is in serialized form of Item which is fundamental unit of data in Sitecore. Because of this we have variety of developer tools which deals with Sitecore Serialization, popular among them is HedgeHog's TDS and Kam's Unicorn.

In early stages of project we use to manual way of serializing the Sitecore Items with help of Sitecore shell itself and de-serialize back into Sitecore using same process..but when we started doing that in bulk people do miss items which they change during development or other human mistakes which lead us to significant time to roll out changes for platform. So, we started using TDS which is convenient for developers(after training) and our item(data) deliverable were in form of update packages.

Another concern was deploying every time full build from code result in huge package which is concern from speed of deployment, as large update and code packages will take more time to deploy & debug if deployment fails. This issue is resolved by simply creating a tag in source control on each successful build and compare your current successful build with given build tag using little magic from PowerShell, PSake,  PowerCore then generate a delta in package. This type of packages we called as Incremental Package. How to create such package would be subject for another article with code snippets.

There are large number of configuration files in Sitecore, for any development project that multiplies when you have multiple deployment environment like SIT, UAT, NNIT(Non Negative Impact testing), PROD etc., But it grows further when you have 10 different production environment running same code but with obvious different configuration. Here a environment means a collection of machines(it could be Sitecore instances too) CM/CD/Job server with there own dedicated databases. So in order to manage this situation there are two solutions one is TDS based environment project where you maintain build profile which mimic your number of environment and than replaces the config from relevant folder(you create copies of config in folder structure per build config) at time of build using appropriate build profile(Like Debug, Release, SIT, UAT etc.,). This approach work marvelously until you are limited to 5-6 configuration targets, but if you are running more than 10 targets then using Config Transformation is best approach and that can work in accordance with Sitecore patch mechanism.

In our deployment architecture we always use three minimum roles of Sitecore Core server first is CM(Content Management/Authoring Node), CD(Content Delivery) and Job Server(All active background jobs) before advent of publishing services and reporting services in Sitecore. Once you have CM/CD/Job split we need to deploy different configs and different data(update packages to different role). For example Code will be deployed to all server subsequently but after testing in CM. Configs will be taken care at deployment time due to use of config transformation technique.
Items, we only need to deploy in CM and that too only once as we have multiple CM servers. Due to this we need our Code and items packages to be separated. We don't compromise with this policy even for public Sitecore Module.
Later after testing Items will be published to first Internal Target to test and further to external target(Live URL for that environment).

Another concern was to analyses of quality of code team is contributing which can be done by employing right tool at time of build. We are using Sonar on same Jenkins server to analyze the code.
Over choice of Jenkins Vs Team City & Bamboo, we did choose Jenkins as economic(Open Source /Free), scale-able, flexible, robust product.

Over choice of multiple source controls we did choose on-premise version of BitBucket because of following reason:

• We use larger suite of Atlassian tools like, for project management we use Jira & Agile Module while for documentation we use Confluence so BitBucket was a natural fit for us.
• Our project policy don't allow us to store code in cloud(SaaS) so we had On-Prem(On premise for us is non public cloud data center) instance of it.
• Our team needed feature branches concept and love the Git-Flow Model, so bit bucket was good fit for us.

Because of policy of no-cloud we do host our own Proget Server to store build artifacts as we wrap our build output even .update file into a Nuget package. This tool also help us to solve in another issue. We want our development team to use only vetted Nuget Packages from open world as we want to control the consistency of dependencies used across the projects. This is possible due feed agregator feature of Proget. Due to proger we are able to host our Sitecore DLLs package which we use in CI-CD process and as project reference for various development project. We also use this tool for our internal community Sitecore Modules pushed as nugget packages. We also convert popular required Sitecore module into Nuget packages.

Last we need a tool which can deploy our Nuget package to running Sitecore instances that at different roles.(CM/CD/Job). So we used Octopus Server which is perfect for .Net based Orchestrator. This allow us to deploy seamlessly and also open other opportunities like Sitecore environment creation automation, Environment Syncing from source, taking code and db backup and restoring whenever we need. We can employ many deployment practices whichever suits to project and team. My colleague Badal has written a detailed series already on how we used Octopus Server here.


Following diagram explain the summary of the Dev-ops process used for our Sitecore Platform.

Figure 2.1

You have notice so far that i am not going in detail for each of these tools and scripts because they are elemental part of it and there are already many articles out which explain already, but in case if you need to know anything specific after reading this article, please let me know, i will go through that in detail. But In next article it will be all about scripts and sample, as in next article we will discuss a new concept of SDM-(Sitecore Dependency Manager) which allows you to create a nugget package which will install all dependencies in server and in local Sitecore instance. This will help in managing same configuration(Dependency + Configuration) at server and at local seamlessly.

Update : Unfortunately Will not be able to release details of SDM as it was designed specifically for Customer and as per their policy can't make it public :(

Dev-Ops with Sitecore

Part-1 | Part-2

Hi Folks,

One of my involvement from last two years was to define Dev-Ops for our platform and in it bits and pieces we evolved it to perfection. Today we may have best in class Dev-Ops(Opinionated) model working but our beginning was humble. Writing this blog series to share our model designed for large Sitecore implementation(Even Multi Sitecore Projects).

But before getting into nitty-gritty of Dev-Ops, Let's understand first what is formal definition of it & goals?

"DevOps (a clipped compound of development and operations) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes." (Source Wikipedia)

Usually Dev-Ops meant different to different people, but for me key goal is to define a end to end process which enabled team to develop and deploy code safely & easily. I can elaborate this line further in following points:

• Ability to release on demand : Team should be focused on Continuous Delivery and the capability of having software always in a releasable state so that when that new feature is ready to go live - it can go live, regardless of where an organization might be in a release cycle. This enabled business release when they like, Predictably, consistently and very fast. In end purpose is to Stop having 'release weekends' - the process becomes like breathing.
• Fail Smart, Recover Fast, Deploy Safely & Consistently : Deployment should not be large block of code and data which replaces every time everything in target environment, it should be small unit which if fails can be recover easily. What you install should be logged so that you know what to recover and before deploying take appropriate backup so that you can return back in time. In end purpose is to reducing the MTTR (Mean Time to Recover/Resolution/Repair - whichever term you prefer).
• Monitor & Measure : Moving to iterative, Agile methodologies enables us to track a feature from inception all the way through to live and measure its cost of travel through the process. Integrating the software manufacturing pipeline and having an effective application monitoring tool that can identify changes at the business transaction level is essential to being able to report back to your stakeholders on the impact of that new piece of code you put into production. Monitoring system where you host application is crucial to identify the issues and mitigate them before they turn into disaster. Over the period of time this monitoring data can be used for further optimization in resources you put to host the application.

Will check upon these goals back & forth as we progress with our series.

Going back to Wikipedia definition, it is clear that there are few stakeholders and few process which make sure that software development and delivery can go hand in hand. Let's discuss stakeholders in context of CMS, usually there are "Developers", "Testers", "Content Authors" and "Deployers".

Developers : People who do the heavy lifting of code and develop features(Usual Sprint Teams). Teams can further classified if project has clear boundaries like who do feature development Vs Hot Fixes, but this or such classification is merely based on scale & complexity of project.

Testers : People who do the testing of release in different testing environment, they could be also external people like business users who want to do User Acceptance testing.

Content Authors : In CMS system major effort goes into content entry or content assembly. Usually such people do interact with only concerned CMS interface but sometime they need to interact with other system for changing things like CSS, JSS ec.,

Deployers : People who click the button and trigger deployment if that button is available. Otherwise getting there hand dirty while manually copy pasting the release to target systems and do various steps in order to make sure that deployment in safe & successful. These people are usually also refereed as IT professionals.

Now we know what are the goals and who are the stakeholders responsible to execute process in order to reach those goals. Now we need to define the process.

Any Dev-Ops process is dependent upon the project, so no single process can become one & only way forward for everyone. For example a small CMS team with limited resource and scope might be happy with decent source control like GitHub and single tool which take care of build and deployment. I have seen and done myself such setup for example GitHub with help of Jenkins or TeamCity combination can do itself everything required out there(Obviously CMS specific tools are counted here). Elsewhere a large project team require different setup like Bitbucket as Source control, Jira Agile for Project Management & Issue Tracking, Jenkins for Build load, Sonar for Code Analysis, ProGet for build repository, Octopus for Deployment.

Point is how much distributed and deeper your Dev-Ops process likely to be depend upon the scale & scope of project or group of projects you want to execute. But process can be largely same even if executed with help of single tool or  multiple tools.

Figure 1.1

In Figure 1.1 Dev-Ops process depicted which started from Jira (Project Management Tool or Agile Management tool) all the way to deployment and monitoring which can trigger incident and circle back to area where these incident can be logged and worked upon by the developers. Each piece is integrated and has a role in overall process. In end of this series will replace this diagram with tools so that it is clear how we do it in end to see larger picture.
Every CMS system required multiple dependencies, for example few could be system dependencies like .NET framework and few could be soft dependencies like extensions or module available for that CMS or other configuration settings which require to be there before your code and content be functional.
So, along with usual Continuous Integration & delivery I will also explain in detail how we can take care of dependency installation in context of CMS systems. We all know deploying your code to existing running environment is straight forward, but installing dependency to that system or preparing a new environment from start is not so straight. Similarly, how to manage dependencies to be used by a project team in day to day development Vs making sure same dependencies exist in server environment.

In order to explain it I will use Sitecore as CMS in further articles, but largely process can be same for other CMS systems too, So watch out for next article.