Archive for the ‘UserProfile’ Category

MySite Deleted by MySite Cleanup Job

Wednesday, May 21st, 2014

Version 1.5

Applied To: SharePoint 2010, 2013, 2016, SharePoint Online

Situation:

Lot of users lost their SharePoint MySite. The site got deleted and they were not able to see any data in their MySite. In some situations, users were prompted for the creation of new MySite. Most of the deleted sites were for inactive users though.

MySite Cleanup Process Chart:

I created an initial version of the following MySite Cleanup Job Process for overview.

SharePoint User Profile MySite Cleanup Job Process by Softvative Inc

Cause:

SharePoint Server has a ‘My Site Cleanup Job’ as a Timer Job. The job runs every hour. You can view the job at this location:

SharePoint Central Administration website > Monitoring > Under ‘Time Jobs’ click ‘Review Job Definitions‘ > Scroll to the bottom of the page and go to next page. Look for ‘My Site Cleanup Job‘ and click on it.

SharePoint My Site Cleanup Job

The function of My Site Cleanup job is to delete the user profile and My site of users that are queued for deletion.

Conditions that will mark the account for deletion:

The user profile and My site will be marked for deletion if any of the following conditions is met.

a. account is deleted in AD (Active Directory)
b. account is disabled in AD
c. SharePoint Profile Sync connection is modified with additional filters, deleted, recreated (e.g filter like exclude user with Department = Terminated)
d. account is moved to an AD OU that is not part of SharePoint Profile Sync

Test Scenario:

I used six test accounts with few of those having managers listed in AD account. The SharePoint Profile Sync added those to the SharePoint profile. Later I logged in as those test users and created their my sites by clicking on the link. Alternatively click username drop down on top right, choose My Site from dropdown. Then click on ‘My Content‘ link on top left.

http://mySiteHost.domain.com/_layouts/MySite.aspx

Then I deleted one, disabled two and moved to a different OU one account. That left the two out of six test accounts as untouched in AD.

Subsequent user profile run marked the accounts based on those conditions with bDeleted =1 in SQL tables. That in layman terms – the site is queued for deletion.

I got following email notification for one of the test account that was setup as my Direct Report in AD accounts properties.

The My Site of Test User 5 is scheduled for deletion in 14 days. As their manager you are now the temporary owner of their site. This temporary ownership gives you access to the site to copy any business-related information you might need. To access the site use this URL: http://mysiteHost.domain.com/personal/testuser5

After 11th day, I got same email but with deletion in 3 days.

SharePoint MySite Cleaner job then looks at the bDeleted=1 and notifies the user’s Manager if one was listed in AD, or Secondary MySite Owner if no manager was configured. It sends second notification after 11 days. After 14 days it deletes the site. In few situations, I’ve seen users were actively working in SharePoint when their My site got deleted. That happens after changes made by AD Team or SharePoint team (conditions above – Conditions that will mark the account for deletion) but after 14 days the site will be deleted even if the changes were reverted.

 

SQL Query to check SharePoint user profiles / My Sites marked for deletion:

I then used the following SQL queries to find the status. It is not a recommended method, use it at your own risk.

1. Use the following query to check the accounts that are marked for deletion. 
— Check the Sharepoint User Profile accounts that are marked for deletion
select * from [Profile DB].[dbo].[UserProfile_Full] with (nolock)
where bDeleted =1
order by PreferredName

2. SQL Query to check the Email Notification Status sent to Manager or Secondary Mysite Owner:
–Query the MysiteDeletion Email Notification status in User Profile DB in Sharepoint
select * from [Profile DB].[dbo].[MySiteDeletionStatus] with (nolock)
–where NotificationStatus = 1 — Email sent to Manager or Seconday contact that site will be deleted in 14 days
— where NotificationStatus = 2 — Email sent to Manager or Seconday contact that site will be deleted in 3 days

3. SQL Query to find the users My Site in MySite Content DB – AllWebs Table
–SQL Queries to check the users under MySite Web App / Content DB
SELECT FullUrl, Title, RequestAccessEmail 
FROM [WSS_Content_MySites].[dbo].[AllWebs]
order by FullUrl

4. SQL Query to check the users info in MySite content DB – UserInfo Table
— Another Query to check users Info under MySite Content DB / MySite Web App
select * from [WSS_Content_MySites].[dbo].[UserInfo] with (nolock)
order by tp_Login

Recommendations:

a. Make sure to test your User Profile Synchronization Connections in pre-production first.

SharePoint Central Admin > Application Management > Manage Service Applications > Click on User Profile Synchronization service application > Click ‘Configure Synchronization Connections’ under Synchronization section. Hover mouse over the connection name, click on drop down, and choose Edit to change the selection of Active Directory OU in the sync. Choose ‘Edit Connection Filters’ from the previous connection page to review / update filters.

SharePoint User Profile Service Application – Synchronization Connection

b. Partially Disable the ‘My Site Cleanup Job’ when editing Profile Sync connection. Later make sure to enable the job. My Site Cleanup Job performs some other cleanups as well like Organization Browser list, People Picker etc.

c. Make sure the ‘Secondary MySite Owner’ account listed has an email address that is monitored by the SharePoint Team or those email get forwarded to SharePoint Team to act on it.

d. Make it as part of SharePoint Governance & training that Managers should notify SharePoint Team when they get the email notification for site deletion with in 14 days.

e. Monitor / control changes done in Active Directory that will impact SharePoint users. Example like moving AD account to a different OU that is not part of share point User Profile Sync.

SharePoint Online / OneDrive for Business

OneDrive for Business stores documents under user’s mysite (part of User Profile) in SharePoint Online (Office 365). Use the following Microsoft KB article for reference:
 
OneDrive for Business retention and deletion – https://support.microsoft.com/en-us/kb/3042522
 
Use OrphanedPersonalSitesRetentionPeriod parameter of Set-SPOTenant PowerShell cmdlet to increase the value of 30 content deletion from default value of 30 days to a higher value. Email notification will still be sent notifying contents will be deleted in 30 day and 7 days. https://technet.microsoft.com/en-us/library/fp161390.aspx
 
Use IncludeOnlyPersonalSite and Limit parameters of Get-SPODeletedSite Powershell cmdlet to get the list of Personal sites (OneDrive for Business contents) that are marked for deletion. https://technet.microsoft.com/library/fp161365.aspx
 
Use SharePoint Online eDiscovery to put a hold on deletion of Mysite (OneDrive For Business) contents. Create a new site collection using eDiscovery template in your SharePoint Online Tenant if you don’t have eDiscovery site in place.
 

Note:
I’ll cover the process on how to analyse and recover the deleted MySites in situations where bulk of Mysites got deleted. That is to see which sites had data and which ones were just not actively used MySites or deleted MySites without any documents to recover. I’ll try to cover the use of PowerShell commands Get-SPDeletedSite and Restore-SPDeletedSite.

References:
1. http://blogs.msdn.com/b/kaevans/archive/2012/06/25/top-recommendations-for-managing-the-my-site-cleanup-timer-job.aspx
2. http://blogs.technet.com/b/fromthefield/archive/2013/07/25/windows-powershell-script-to-output-site-collection-information.aspx

Search Keywords:

MySite Deleted by SharePoint User Profile MySite Cleanup Job
MySite Deleted by MySite Cleanup Job
MySite Cleanup Job
User Profile Sync removed mysite
SharePoint MySite Removed
SharePoint MySite deleted
SharePoint User Profile deleted
How A User Profile is Deleted?
How A MY Site is Deleted?
My Site deletion Mystery
Process of My Site Deletion
My Site Deletion Process
Lost MySite Data
Lost SharePoint Mysite
Onedrive for Business deletion
Prevent OneDrive for Business deletion
Retain OneDrive for Business contents
Prevent SharePoint Online user site deletion
SharePoint Online mysite deletion
SharePoint eDiscovery to prevent OneDrive Deletion

Sharepoint Personal MySite Quota Limit Reached

Thursday, May 8th, 2014

Scenerio:

The default personal MySite storage limit in Sharepoint 2010 is 100 MB with warning email sent after 80 MB. A user exceeded the default MySite storage limit of 100 MB. When changing his Mysite permissions user got the following error:

Error:
Your changes could not be saved because the SharePoint web site has exceeded the storage quota limit.
You must save your work to another location. contact your administrator to change the quota limits for the web site.
Correlation ID:
Date and Time:

SharePoint User’s Personal MySite Storage limit reached Error

Resolution:

I increased the Quota limit of personal MySites to 250 MB with Warning email notification to user at 200 MB. The new quota  template was applied to user’s Mysite.

Resolution that was followed:
1. To update Quota Template limit, go to SharePoint Central  Administration website
2. Click on Application Management > Specify Quota Template (under Site Collection section)
3. Edit the ‘Personal Site’ Template (that is applied by default to mySites). Enter the limits for maximum storage and warning email under ‘Storage Limit Values’ section
4. Click OK

This only updates the quota Template limit and doesn’t actually apply it to the users’ mysites. Now apply the quota template to Mysites that are having issue using following steps:

1. Go to Central Admin > Application Management
2. Click on ‘Configure quotas and locks’ under Site collection section
3. Select the user’s site collection from the drop down by selecting the Mysite web application and user’s Mysite Site Collection.
4. Under ‘Site Quota Information’ section select the ‘Personal Quota’. Make sure it shows the updated limits (grayed out).
5. Click Ok

That single user’s Mysite has the updated quota applied to it. The other Mysites of users will still have the previous 100 MB limit until you apply the quota template on those MySites again.


Keywords:

  • SharePoint mysite
  • SharePoint mysite quota
  • check storage quota sharepoint mysite
  • SharePoint MySite Quota limit
  • MySite Quota limit
  • MySite Storage Quota limit
  • SharePoint MySite storage Quota

MOSS MA Not Found error UserProfile Connection

Friday, April 25th, 2014
In SharePoint 2010, you want to setup User Profile Connection to Active Directory. You configure everything and click OK and it brings error message.

Issue Scenario:

  1. Go to SharePoint Central Administration website
  2. Click on ‘Manage Service Applications’ under Application Management
  3. Click on ‘User Profile Service Application’ or the name of your user profile service app
  4. On ‘Manage Profile Service’ page, click on ‘Configure Synchronization Connections’ under Synchronization section
  5. Click ‘Create New Connection’
  6. Enter the Connection Name, Forest Name, Account Name, Password, select OU levels under Containers section to sync profiles from that OU level
  7. Click OK to configure the connection

In this case, you will get error:

Error

MOSS MA Not Found
Troubleshoot Issues with Microsoft Sharepoint Foundation
Correlation ID: f8224450-ea42-742a-d8c6-5de674235e32

Date and Time: MM/DD/YYYY hh:mm:ss AM
Go back to site
MOSS MA Not Found Error
Finding the Correlation ID Error Details:
I had the following Codeplex tool deployed that helps in searching the ULS logs for the Correlation ID details. The tool is available under Central Administration website > Monitoring > Query correlation ID (under Logging section). You enter the Correlation ID from the error page, specify a date / time range and it shows the results.
That tool provided the following query result on its page. Looking at the query details, I saw the highlighted line. That section was telling that it could not connect to App Server 1 at port 5725. That server at this point was not configured for the User Profile Service. The service was setup on App Server 3.
Correlation Query Result:
Time Process Area Category Level EventID Message 
04/2/2014 10:14:38.90  w3wp.exe (SrvApp03:0x1C6C)  SharePoint Foundation  Logging Correlation Data  Medium xmnv Name=Request 
(POST:http://SrvApp03:19999/_layouts/EditDSServer.aspx?ApplicationID=abb4158a%2De70a%2D4185%2D86e2%2Df02f1c25966e) 
04/2/2014 10:14:38.90  w3wp.exe (SrvApp03:0x1C6C)  SharePoint Foundation  Logging Correlation Data  Medium xmnv Site=/ 
04/2/2014 10:14:39.97  w3wp.exe (SrvApp03:0x1C6C)  SharePoint Portal Server  User Profiles  High d3b3 LoadConnections failed trying to fill the connections list. Most likely during RetriveResources because of permissions — {1}. Available parameters: 
System.ServiceModel.EndpointNotFoundException: Could not connect to http://SrvApp01:5725/ResourceManagementService/MEX. TCP error code 10061: No connection could be made because the target machine actively refused it 10.1.0.103:5725. —> 

System.Net.WebException: Unable to connect to the remote server —> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.1.0.103:5725 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, 

SocketAddress socketAddress) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& 
… 
04/2/2014 10:14:39.97* w3wp.exe (SrvApp03:0x1C6C)  SharePoint Portal Server  User Profiles  High d3b3 …socket, IPAddress& address, 

ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception) — End of inner exception stack trace — 

at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context) at System.Net.HttpWebRequest.GetRequestStream() at 

System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream() — End of inner exception stack trace — Server 

stack trace: at System.ServiceModel.Channels.HttpOutput.WebRequestHttpOutput.GetOutputStream() at 

System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout) at 

System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout) 

at System.ServiceModel.Channels.RequestChannel.Req… 
04/2/2014 10:14:39.97* w3wp.exe (SrvApp03:0x1C6C)  SharePoint Portal Server  User Profiles  High d3b3 …uest(Message message, 

TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) at 

System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, 

Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, 

ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at 

[0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at 

System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at 

System.ServiceModel.Description.IMetad… 
04/2/2014 10:14:39.97* w3wp.exe (SrvApp03:0x1C6C)  SharePoint Portal Server  User Profiles  High d3b3 …ataExchange.Get(Message 

request) at Microsoft.ResourceManagement.WebServices.MetadataClient.Get(String dialect, String identifier) at 

Microsoft.ResourceManagement.WebServices.Client.ResourceManagementClient.SchemaManagerImplementation.RefreshSchema() at 

Microsoft.ResourceManagement.WebServices.ResourceManager.get_SchemaManager() at 

Microsoft.ResourceManagement.WebServices.ResourceManager..ctor(String typeName, LocaleAwareClientHelper localePreferences, 

ContextualSecurityToken securityToken) at Microsoft.Office.Server.UserProfiles.ConnectionManager.LoadConnections(Boolean fForUI) . 
04/2/2014 10:14:39.97  w3wp.exe (SrvApp03:0x1C6C)  SharePoint Portal Server  User Profiles  High a3xu 

ConnectionManager.LoadConnections(): Could not find MOSS MA despite being marked as fully configured, was it deleted? 
04/2/2014 10:14:40.09  w3wp.exe (SrvApp03:0x1C6C)  SharePoint Foundation  Monitoring  Medium b4ly Leaving Monitored Scope (Request 

(POST:http://SrvApp03:8000/_layouts/EditDSServer.aspx?ApplicationID=abb4158a%2De70a%2D4185%2D86e2%2Df02f1c25966e)). Execution Time=1193.10918007736 
Correlation Query Tool’s Result based on Correlation ID Error
Troubleshooting and Resolution:
  1. Looking at the query results, I saw that connection was trying to connect to first server and connection was refused. That server has been having issues and was already planned for removal from the farm. It was not running the User Profile Service.
  2. I then looked at the ‘Manage Servers in the Farm‘ page in SharePoint Central Administration > System Settings. I looked for ‘User Profile‘ service on that page to confirm which servers in the farm are running that service. It was not configured on App Server 1. I then went to ‘Manage Service Applications‘ page under Application Management in Central Administration Website. 
    SharePoint Manage Service Applications Page – Properties icon grayed out 

     Hover over the ‘User Profile Service Application‘, don’t click on the name but click on the right side of it. That will select the service application and activate the Operations Ribbon group on top. 

    SharePoint Manage Service Applications Page – Properties icon active

     

  3. Click on Properties icon in the SharePoint ribbon. That will bring the ‘Edit User Profile Service Application‘ dialog box. Scroll to the lower half of page and look under ‘Profile Synchronization Instance‘ field. It is fourth field from the bottom. In my case, it was set to the App Server 1. I changed it to the server that was actively running User Profile Service and clicked OK at the bottom.
    SharePoint – User Profile Service Application – Profile Synchronization Instance – Old Server

     

  4. Message was displayed saying ‘Profile Service Application successfully updated.’ Click ok on that dialog box to close it.
  5. I tried to configure the Profile Synchronization Connection again (Steps in Issue Scenario section above). Same error happened. I looked at the ‘Profile Synchronization Instance’ field (step 3 above), the same old App Server 1 was listed there. Not good.
  6. I started the ‘User Profile Service’ on that old App Server 1 from the Central Administration > ‘Manage Services on Server’ page for App Server 1.
    Sharepoint – Started User Profile Service from Manage Services on Server page

      

  7. I then changed the server under ‘Profile Synchronization Instance‘ to a new server that was running the User Profile Service. This time the setting stayed permanently and didn’t revert back as in last attempt.
    SharePoint – User Profile Service Application – Profile Synchronization Instance – Active Server

      

  8. I tried to configure user profile sync again. (Steps in Issue Scenario section above). It was successful this time.
Important Thing:
One important thing. I didn’t have to start the ‘Forefront Identity Manager Service‘ on the old server using Windows Services console. That service was in disabled state as before. After successfull Sync connection, I then stopped the ‘User Profile Service‘ on the App Server 1 that was started in step 6 above. I used SharePoint Central Administration website. It is not recommended to start / stop FIM service using  Windows Services console.

Sharepoint – Forefront Identity Manager Service (part of User Profile Service App) in Windows Services Console

References:
1. http://blogs.msdn.com/b/sofocle/archive/2011/06/28/user-profile-syncronization-failure-quot-moss-ma-not-found-quot.aspx
2. http://manojviduranga.wordpress.com/2013/05/18/moss-ma-not-found-when-creating-a-new-connection-to-aduser-profile-synchronization-in-sharepoint-2010/
3. http://mohamedelkassas.wordpress.com/2013/03/15/moss-ma-not-found-user-profile-syncronization-failure/
4. http://social.technet.microsoft.com/Forums/sharepoint/en-US/cf35f233-c6a9-440a-aa61-8be190782fcc/user-profile-synchronization-error-moss-ma-not-found
5. http://social.msdn.microsoft.com/Forums/en-US/81684a0f-30a0-4a29-b433-54e382666f47/user-profile-services-sync-connection-disappearing-after-a-few-days-moss-ma-not-found?forum=sharepointadminprevious

Sharepoint 2010 Timer Service Stops Unexpectedly

Thursday, March 27th, 2014
Issue:
SharePoint 2010 Timer Service Stops Unexpectedly / SharePoint 2010 Timer Service keeps failing. Starting the service in Windows Services console will fail after sometime , logging error in Event Viewer.

Environment: 
SharePoint Application Server in a multi-tier medium farm. SharePoint WFE, App and database running on separate servers.

Scenario:
On one of the app server SharePoint 2010 Timer service keeps failing. In Windows Services console window if you start the service, after some moment it will stop automatically.

Looking at the Event Viewer logs on the server reports the following errors.

Error 1 on App Server 1
Log Name:      System
Source:        Service Control Manager
Date:          1/25/2014 11:28:04 AM
Event ID:      7034
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SPAppServer1.domain.com
Description:
The SharePoint 2010 Timer service terminated unexpectedly.  It has done this 4 time(s).
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Service Control Manager” Guid=”{555908d1-a6d7-4695-8e1e-26931d2012f4}” 

EventSourceName=”Service Control Manager” />
    <EventID Qualifiers=”49152″>7034</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-25T16:28:04.344856500Z” />
    <EventRecordID>144167</EventRecordID>
    <Correlation />
    <Execution ProcessID=”524″ ThreadID=”6932″ />
    <Channel>System</Channel>
    <Computer>SPAppServer1.domain.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name=”param1″>SharePoint 2010 Timer</Data>
    <Data Name=”param2″>4</Data>
  </EventData>
</Event>

Error 2 on App Server 1
Log Name:      System
Source:        Service Control Manager
Date:          1/25/2014 11:25:04 AM
Event ID:      7031
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SPAppServer1.domain.com
Description:
The SharePoint 2010 Timer service terminated unexpectedly.  It has done this 2 time(s).  The 

following corrective action will be taken in 30000 milliseconds: Restart the service.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Service Control Manager” Guid=”{555908d1-a6d7-4695-8e1e-26931d2012f4}” 

EventSourceName=”Service Control Manager” />
    <EventID Qualifiers=”49152″>7031</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-25T16:25:04.378034500Z” />
    <EventRecordID>144161</EventRecordID>
    <Correlation />
    <Execution ProcessID=”524″ ThreadID=”848″ />
    <Channel>System</Channel>
    <Computer>SPAppServer1.domain.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name=”param1″>SharePoint 2010 Timer</Data>
    <Data Name=”param2″>2</Data>
    <Data Name=”param3″>30000</Data>
    <Data Name=”param4″>1</Data>
    <Data Name=”param5″>Restart the service</Data>
  </EventData>
</Event>

Error 3 on App Server 1
Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          1/26/2014 11:55:00 AM
Event ID:      8312
Task Category: Topology
Level:         Information
Keywords:      
User:          domainSPFarmSvcAct
Computer:      SPAppServer1.domain.com
Description:
SharePoint Web Services Round Robin Service Load Balancer Event: Initialization
Process Name: OWSTIMER
Process ID: 5132
AppDomain Name: DefaultDomain
AppDomain ID: 1
Service Application Uri: urn:schemas-microsoft-

com:sharepoint:service:abb4158ae70a418586e2f02f1c25966e#authority=urn:uuid:db81a38b0d1f41a0b81887

9a36c1c4cf&authority=https://SPAppServer2:32844/Topology/topology.svc
Active Endpoints: 1
Failed Endpoints:0
Endpoint List:
http://SPAppServer1:32843/abb4158ae70a418586e2f02f1c25966e/ProfileService.svc

Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Microsoft-SharePoint Products-SharePoint Foundation” Guid=”{6FB7E0CD-52E7-

47DD-997A-241563931FC2}” />
    <EventID>8312</EventID>
    <Version>14</Version>
    <Level>4</Level>
    <Task>13</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-26T16:55:00.166105300Z” />
    <EventRecordID>560552</EventRecordID>
    <Correlation ActivityID=”{A04B162B-0875-4979-9829-C2055D7BFC99}” />
    <Execution ProcessID=”5132″ ThreadID=”6840″ />
    <Channel>Application</Channel>
    <Computer>SPAppServer1.domain.com</Computer>
    <Security UserID=”S-1-5-21-1627688274-1190192956-3999157559-1626″ />
  </System>
  <EventData>
    <Data Name=”string0″>Initialization</Data>
    <Data Name=”string1″>OWSTIMER</Data>
    <Data Name=”int2″>5132</Data>
    <Data Name=”string3″>DefaultDomain</Data>
    <Data Name=”int4″>1</Data>
    <Data Name=”string5″>urn:schemas-microsoft-

com:sharepoint:service:abb4158ae70a418586e2f02f1c25966e#authority=urn:uuid:db81a38b0d1f41a0b81887

9a36c1c4cf&amp;authority=https://SPAppServer2:32844/Topology/topology.svc</Data>
    <Data Name=”int6″>1</Data>
    <Data Name=”int7″>0</Data>
    <Data 

Name=”string8″>http://SPAppServer1:32843/abb4158ae70a418586e2f02f1c25966e/ProfileService.svc
</Data>
  </EventData>
</Event>
Error 4 – DCOM 10016 on App Server 1
Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          1/25/2014 8:37:42 AM
Event ID:      10016
Task Category: None
Level:         Error
Keywords:      Classic
User:          NETWORK SERVICE
Computer:      SPAppServer1.domain.com
Description:
The application-specific permission settings do not grant Local Launch permission for the COM 

Server application with CLSID 
{1BE1F766-5536-11D1-B726-00C04FB926AF}
 and APPID 
{1BE1F766-5536-11D1-B726-00C04FB926AF}
 to the user NT AUTHORITYNETWORK SERVICE SID (S-1-5-20) from address LocalHost (Using LRPC). 

This security permission can be modified using the Component Services administrative tool.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Microsoft-Windows-DistributedCOM” Guid=”{1B562E86-B7AA-4131-BADC-

B6F3A001407E}” EventSourceName=”DCOM” />
    <EventID Qualifiers=”49152″>10016</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-25T13:37:42.000000000Z” />
    <EventRecordID>143120</EventRecordID>
    <Correlation />
    <Execution ProcessID=”0″ ThreadID=”0″ />
    <Channel>System</Channel>
    <Computer>SPAppServer1.domain.com</Computer>
    <Security UserID=”S-1-5-20″ />
  </System>
  <EventData>
    <Data Name=”param1″>application-specific</Data>
    <Data Name=”param2″>Local</Data>
    <Data Name=”param3″>Launch</Data>
    <Data Name=”param4″>{1BE1F766-5536-11D1-B726-00C04FB926AF}</Data>
    <Data Name=”param5″>{1BE1F766-5536-11D1-B726-00C04FB926AF}</Data>
    <Data Name=”param6″>NT AUTHORITY</Data>
    <Data Name=”param7″>NETWORK SERVICE</Data>
    <Data Name=”param8″>S-1-5-20</Data>
    <Data Name=”param9″>LocalHost (Using LRPC)</Data>
  </EventData>
</Event>
A quick search on Microsoft KB article resulted in the following solution for the DCOM issue
At the 4th step of above KB article, I got stuck with the security tab being grayed out. I was running Windows 2008 R2. In short you take ownership for the service account and add full control permissions of the following registry key:
HKEY_CLASSES_ROOTAppID{61738644-F196-11D0-9953-00C04FD919C1}
Default Settings
Farm Service Account with permissions added
Reference: 
I found the following article to enable the security tab for IIS WAMREG Admin. 
After following above, the error 10016 was gone. I started the Sharepoint 2010 Timer Service but it stopped again after some time reporting the same errors listed above (except DCOM 10016 error) . A review of Central Admin app server, following errors were seen in the Event Viewer.
Error 1 on Central Admin App Server
Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          1/26/2014 11:20:29 AM
Event ID:      8313
Task Category: Topology
Level:         Error
Keywords:      
User:          domainFMasood
Computer:      SPAppServer2.domain.com
Description:
SharePoint Web Services Round Robin Service Load Balancer Event: EndpointFailure
Process Name: w3wp
Process ID: 9888
AppDomain Name: /LM/W3SVC/784624454/ROOT-1-130403119376795973
AppDomain ID: 2
Service Application Uri: urn:schemas-microsoft-

com:sharepoint:service:de6d45dc853941058fe7920ad771208a#authority=urn:uuid:db81a38b0d1f41a0b81887

9a36c1c4cf&authority=https://SPAppServer1:32844/Topology/topology.svc
Active Endpoints: 2
Failed Endpoints:1
Affected Endpoint: http://SPWFEServer1:32843/de6d45dc853941058fe7920ad771208a/SearchService.svc
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Microsoft-SharePoint Products-SharePoint Foundation” Guid=”{6FB7E0CD-52E7-

47DD-997A-241563931FC2}” />
    <EventID>8313</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>13</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-26T16:20:29.100686100Z” />
    <EventRecordID>1953825</EventRecordID>
    <Correlation ActivityID=”{CF33969D-E8B0-47C9-8A64-666110FB9663}” />
    <Execution ProcessID=”9888″ ThreadID=”9660″ />
    <Channel>Application</Channel>
    <Computer>SPAppServer2.domain.com</Computer>
    <Security UserID=”S-1-5-21-1627688274-1190192956-3999157559-386822″ />
  </System>
  <EventData>
    <Data Name=”string0″>EndpointFailure</Data>
    <Data Name=”string1″>w3wp</Data>
    <Data Name=”int2″>9888</Data>
    <Data Name=”string3″>/LM/W3SVC/784624454/ROOT-1-130403119376795973</Data>
    <Data Name=”int4″>2</Data>
    <Data Name=”string5″>urn:schemas-microsoft-

com:sharepoint:service:de6d45dc853941058fe7920ad771208a#authority=urn:uuid:db81a38b0d1f41a0b81887

9a36c1c4cf&amp;authority=https://SPAppServer1:32844/Topology/topology.svc</Data>
    <Data Name=”int6″>2</Data>
    <Data Name=”int7″>1</Data>
    <Data 

Name=”string8″>http://SPWFEServer1:32843/de6d45dc853941058fe7920ad771208a/SearchService.svc</Data

>
  </EventData>
</Event>
The following error(s) were noted on the SharePoint Web Front End (WFE)
Error 1 on WFE
Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Portal Server
Date:          1/26/2014 11:00:04 AM
Event ID:      5553
Task Category: User Profiles
Level:         Error
Keywords:      
User:          domainSPFarmSvcAct
Computer:      SPWFEServer1.domain.com
Description:
failure trying to synch site 96372276-8424-4f7a-9796-b05091a9f2ab for ContentDB 1146e13b-cb37-

43ab-83ce-cbc2e9d34248 WebApp 615621ff-80ea-487a-8ee5-60a7baaea6d4.  Exception message was Cannot 

insert duplicate key row in object ‘dbo.UserMemberships’ with unique index 

‘CX_UserMemberships_RecordId_MemberGroupId_SID’. The duplicate key value is (0c37852b-34d0-418e-

91c6-2ac25af4be5b, 9851, 600, 0x010500000000000515000000528d04613ce7f046374d5eeea1630000).
The statement has been terminated..
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Microsoft-SharePoint Products-SharePoint Portal Server” Guid=”{8B3DDD3D-2B09

-4669-BF81-E2D6921FEEEA}” />
    <EventID>5553</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>8</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-26T16:00:04.103198800Z” />
    <EventRecordID>3208869</EventRecordID>
    <Correlation ActivityID=”{406758CA-3384-438B-936D-75C531CE9928}” />
    <Execution ProcessID=”2176″ ThreadID=”4584″ />
    <Channel>Application</Channel>
    <Computer>SPWFEServer1.domain.com</Computer>
    <Security UserID=”S-1-5-21-1627688274-1190192956-3999157559-1626″ />
  </System>
  <EventData>
    <Data Name=”string0″>failure trying to synch site 96372276-8424-4f7a-9796-b05091a9f2ab for 

ContentDB 1146e13b-cb37-43ab-83ce-cbc2e9d34248 WebApp 615621ff-80ea-487a-8ee5-

60a7baaea6d4.</Data>
    <Data Name=”string1″>Cannot insert duplicate key row in object ‘dbo.UserMemberships’ with 

unique index ‘CX_UserMemberships_RecordId_MemberGroupId_SID’. The duplicate key value is 

(0c37852b-34d0-418e-91c6-2ac25af4be5b, 9851, 600, 

0x010500000000000515000000528d04613ce7f046374d5eeea1630000).
The statement has been terminated.</Data>
  </EventData>
</Event>
Error 2 on WFE
Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          1/26/2014 11:00:00 AM
Event ID:      2137
Task Category: Health
Level:         Error
Keywords:      
User:          domainSPFarmSvcAct
Computer:      SPWFEServer1.domain.com
Description:
The SharePoint Health Analyzer detected an error.  One or more servers is not responding.
The following servers have not executed any timer jobs in the last 2 hours: SPAppServer1.  This 

can happen if the server was shut down or lost network connectivity, or if the timer service on 

that server has crashed, hung, or been stopped.
Ensure that the server(s) listed above are running and connected to the network.  If the timer 

service is not started, restart the service by typing the following command at the command prompt 

on each server: “net start SPTimerV4.”  If the server was intentionally removed from the farm, 

remove the record of the server from the SharePoint topology in the Central Administration site 

at http://SPAppServer2:8000/_admin/FarmServers.aspx. For more information about this rule, see 

“http://go.microsoft.com/fwlink/?LinkID=142656”.

Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Microsoft-SharePoint Products-SharePoint Foundation” Guid=”{6FB7E0CD-52E7-

47DD-997A-241563931FC2}” />
    <EventID>2137</EventID>
    <Version>14</Version>
    <Level>2</Level>
    <Task>8</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-26T16:00:00.931303500Z” />
    <EventRecordID>3208868</EventRecordID>
    <Correlation ActivityID=”{BC7B69CF-8A4C-4E48-8AF8-87CED41958A0}” />
    <Execution ProcessID=”2176″ ThreadID=”2652″ />
    <Channel>Application</Channel>
    <Computer>SPWFEServer1.domain.com</Computer>
    <Security UserID=”S-1-5-21-1627688274-1190192956-3999157559-1626″ />
  </System>
  <EventData>
    <Data Name=”string0″>One or more servers is not responding.
The following servers have not executed any timer jobs in the last 2 hours: SPAppServer1.  This 

can happen if the server was shut down or lost network connectivity, or if the timer service on 

that server has crashed, hung, or been stopped.
Ensure that the server(s) listed above are running and connected to the network.  If the timer 

service is not started, restart the service by typing the following command at the command prompt 

on each server: “net start SPTimerV4.”  If the server was intentionally removed from the farm, 

remove the record of the server from the SharePoint topology in the Central Administration site 

at http://SPAppServer2:8000/_admin/FarmServers.aspx. For more information about this rule, see 

“http://go.microsoft.com/fwlink/?LinkID=142656”.
</Data>
  </EventData>
</Event>
Stop the User Profile Service in Sharepoint Central Admin
The user profile error on the WFE prompted me to stop the User Profile Service on the App Server 1 and enable it on other server. I went through the following to start ‘User Profile Service’ on App Server 2 and stop it on App Server 1.
1. On Central Administration homepage click on ‘Manage servers in the farm’ under Systems Settings section
2. Click on an application server name that can serve the ‘User Profile Service’
3. Under Action column for the User Profile Service, click Start to start the service. Refresh the page after sometime to make sure the status column says Started
4. Go back to Page in step 2 and follow below
5. Click on the application server that has Sharepoint 2010 Timer Service stopping automatically and is running the ‘User Profile Service’
6. On that server, click Stop for the service under Action column
7. Refresh the page to make sure the service is stopped 

Start the ‘User Profile Service’ on App Server 2 using Central Admin
Now the service is started on App Server 2 and stopped on App Server 1 from Central Administration. Review the Windows Services console for the the service called ‘Forefront Identity Manager Service’.
After stopping the User Profile Service from Central Admin, Windows Event Viewer reported the following errors. It seems like the service had some issues and that why it crashed the process.
Error on App Server 1 – Windows Error Reporting
Log Name:      Application
Source:        Windows Error Reporting
Date:          1/26/2014 2:22:03 PM
Event ID:      1001
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      SPAppServer1.domain.com
Description:
Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: owstimer.exe
P2: 14.0.7105.5000
P3: 51e754a5
P4: mscorlib
P5: 2.0.0.0
P6: 5265c965
P7: 4220
P8: 51
P9: DI4EQZGVCWKDLTHKQTZWPXE3YRFKN1I2
P10: 

Attached files:

These files may be available here:
C:ProgramDataMicrosoftWindowsWERReportQueue

AppCrash_owstimer.exe_9a66a4341b3e295f2340a8bc2dbe53c6c5_440cadbf6144

Analysis symbol: 
Rechecking for solution: 0
Report Id: 44e8fbeb81-b51b-4411e3-9abb-44005056930138
Report Status: 4
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Windows Error Reporting” />
    <EventID Qualifiers=”0″>1001</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-26T19:22:03.000000000Z” />
    <EventRecordID>560580</EventRecordID>
    <Channel>Application</Channel>
    <Computer>SPAppServer1.domain.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>CLR20r3</Data>
    <Data>Not available</Data>
    <Data>0</Data>
    <Data>owstimer.exe</Data>
    <Data>14.0.7105.5000</Data>
    <Data>51e754a5</Data>
    <Data>mscorlib</Data>
    <Data>2.0.0.0</Data>
    <Data>5265c965</Data>
    <Data>4220</Data>
    <Data>51</Data>
    <Data>DI4EQZGVCWKDLTHKQTZWPXE3YRFKN1I2</Data>
    <Data>
    </Data>
    <Data>
    </Data>
    <Data>C:ProgramDataMicrosoftWindowsWERReportQueue

AppCrash_owstimer.exe_9a66a4341b3e295f2340a8bc2dbe53c6c5_440cadbf61</Data>
    <Data>
    </Data>
    <Data>0</Data>
    <Data>e8fbeb81-b51b-11e3-9abb-44005056930138</Data>
    <Data>4</Data>
  </EventData>
</Event>
Error on App Server 1 – WAS
Log Name:      System
Source:        Microsoft-Windows-WAS
Date:          1/26/2014 2:53:43 PM
Event ID:      5009
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SPAppServer1.domain.com
Description:
A process serving application pool ‘dab7bc212ce84107bf62c8e661155b7c’ terminated unexpectedly. 

The process id was ‘5220’. The process exit code was ‘0xfffffffe’.
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Microsoft-Windows-WAS” Guid=”{524B5D04-133C-4A62-8362-64E8EDB9CE40}” 

EventSourceName=”WAS” />
    <EventID Qualifiers=”32768″>5009</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-26T19:53:43.000000000Z” />
    <EventRecordID>144299</EventRecordID>
    <Correlation />
    <Execution ProcessID=”0″ ThreadID=”0″ />
    <Channel>System</Channel>
    <Computer>SPAppServer1.domain.com</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name=”AppPoolID”>dab7bc212ce84107bf62c8e661155b7c</Data>
    <Data Name=”ProcessID”>5220</Data>
    <Data Name=”ExitCode”>fffffffe</Data>
  </EventData>
</Event>
Outcome:
At this point, the User Profile Service was active on App Server 2 and was stopped on App Server 1. After it starting the Sharepoint 2010 Timer Service on the App Server 1 worked and the service stayed active.
Error on WFE Server
Log Name:      Application
Source:        Microsoft-SharePoint Products-SharePoint Foundation
Date:          1/26/2014 6:10:30 AM
Event ID:      6398
Task Category: Timer
Level:         Critical
Keywords:      
User:          domainSPFarmSvcAct
Computer:      SPWFEServer1.domain.com
Description:
The Execute method of job definition 

Microsoft.SharePoint.Administration.SPTimerRecycleJobDefinition (ID b88100e2-5681-4b05-929c-

34926b8216a4) threw an exception. More information is included below.

The timer service was not recycled because the following jobs were still running: Microsoft 

SharePoint Foundation Usage Data Import
Event Xml:
<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event”>
  <System>
    <Provider Name=”Microsoft-SharePoint Products-SharePoint Foundation” Guid=”{6FB7E0CD-52E7-

47DD-997A-241563931FC2}” />
    <EventID>6398</EventID>
    <Version>14</Version>
    <Level>1</Level>
    <Task>12</Task>
    <Opcode>0</Opcode>
    <Keywords>0x4000000000000000</Keywords>
    <TimeCreated SystemTime=”2014-01-26T11:10:30.850325300Z” />
    <EventRecordID>3208858</EventRecordID>
    <Correlation ActivityID=”{88AD9857-12B9-421C-AE8A-C2A98C0D4204}” />
    <Execution ProcessID=”2176″ ThreadID=”1504″ />
    <Channel>Application</Channel>
    <Computer>SPWFEServer1.domain.com</Computer>
    <Security UserID=”S-1-5-21-1627688274-1190192956-3999157559-1626″ />
  </System>
  <EventData>
    <Data Name=”string0″>Microsoft.SharePoint.Administration.SPTimerRecycleJobDefinition</Data>
    <Data Name=”string1″>b88100e2-5681-4b05-929c-34926b8216a4</Data>
    <Data Name=”string2″>The timer service was not recycled because the following jobs were still 

running: Microsoft SharePoint Foundation Usage Data Import</Data>
  </EventData>
</Event>
Follow this article for the above error. 
Now after this work, make sure to review the Event Viewer log for any errors. Test the User Profile Service functionality by searching for the users and visiting the user profile pages.
Search and visit user’s Profile to Test the working of User Profile Service

Enjoy SharePoint!