Skip to end of metadata
Go to start of metadata

01/02/2018:  8:00AM

Marked up files in Crocodoc were migrated to Box View.

12/31/2017:  2:30AM to 7:36AM

BbLearn was upgraded from CU4 (v2017 Q2) to CU5 in order to accomodate the migration from Crocodoc to Box View.


12/02/2017:  1:31AM to 3:00AM

Blackboard requested an "audit fix" to our BbLearn instance which required down time.  This was in response to BbSupport ticket #  02990254.

09/18/2017:  3:10PM (estimated)  to  9/19/2017:  8:38AM

Due to networking issues at BbLearn's VA2 and VA3 data centers, slow response times were experienced with colloborate, assignment submissions and access, Safe Assign and Crocodoc.  As BbSupport was somehow unaware of these networking problems, NAU's BbLearn cluster was restarted twice, using a rolling restart both times.  Although the second rolling restart wasn't completed until 09/19 (8:38AM), the affected tools were probably available earlier, as anecdotal evidence from other managed hosted clients indicate that the networking issues were resolved earlier.

09/10/2017:  1:30AM to 2:52AM

BbLearn building block "!NAU F2S Connector," a new version of the older "!NAU GPS Connector," was installed.  Building block "iClicker" was updated to v5.3.9.  After installation and updating, a rolling restart of all nodes was performed.

08/19/2017:  1:31AM to 12:22PM

BbLearn upgraded to v2017 Q2 with CU2.  This major upgrade required a service interruption.  Users were unable to access the system during the timeframe listed.

06/24/2017:  1:30AM to 3:31AM

Rolling restart to refresh nodes after updates to Bb Mobile Web Services and Partner Cloud building blocks


06-08 - 06/09/2017:  7:00PM to 3:00AM

Bb Managed Hosting Data Center maintenance:  no service interruptions were anticipated, but they "cannot entirely rule out brief periods of intermittent connectivity as network traffic adjusts, or quick restart of services on a few client systems if warranted."

06/08/2017:  1:30AM to 3:08AM

Rolling restart to refresh nodes in hopes of clearing up backlogged SIS integration feed transactions.


05/06/2017:  1:30AM to 3:36AM

Rolling restart to refresh nodes prior to start of final exam week.

04/22/2017:  1:41AM to 3:07AM

Rolling restart after installing and then removing the NBC Building Block.

04/01/2017:  1:30AM to 3:25AM

Learn Migration Information Gathering building block installed, rolling restart needed to complete the install.


03/18/2017:  1:30AM to 5:05AM

Blackboard Learn was successfully upgraded to CU4.  CU4 release note are here:


01/31/2017:  2:01AM to 4:39AM

U of York building block installed, rolling restart to needed to complete the install.


01/21/2017:  1:30AM to 3:30AM

Rolling restart to complete the removal of the UserVoice "Feedback" tab and the configuration of an LTI integration for "REEF", which is a polling tool.


11/19/2016:  1:00AM to 4:00AM

Rolling restart to complete the upgrade of the Kaltura and Lynda building blocks.  

11/12/2016:  12:30AM to 4:31AM

Orace was successfully upgraded to v12.1.0.2.  This required down time

10/15/2016:  12:30AM to 11:15AM

Blackboard Learn was successfully upgraded to CU3.  The CU3 upgrade was actually completed by 5:30AM, however a misconfiguration on Bb Manage Hosting's part pointed our CAS login to OWU (Ohio Wesleyan University), true story.  This was corrected by managed hosting which was followed by an rolling restart.  All service was restored by around 11:15AM.


10/08/2016:  1:30AM to 4:30AM

Rolling restart to address instances in which instructors cannot edit Group Set Memberships in Groups sets.  (BbSupport case #02373838).


09/24/2016:  1:00AM to 3:07AM

Rolling restart to complete the installation of the ExamSoft building block


09/17/2016:  12:30AM to 6:36AM

BbLearn off line to install CU2 to 2016 Q2.  The Kaltura (v5.0.51) and the Content Editor (v3000.1.2-rel.16+e53c5e2) building blocks were also updated.


09/03/2016:  1:02AM to  2:15AM

Rolling restart to complete the installation of the McGraw-Hill building block


08/22/2016:  2:00PM to  4:18PM

New versions of the "Pearson Custom Integration" (v3.0.201602144) and "!NAU GPS Connector" building blocks were installed followed by a rolling restart.

08/20/2016:  10:00PM to 08/21/2016:  9:20AM

Blackboard Learn was upgraded to 2016 Q2 with Cumulative Update 1.  All building blocks on the system were also brought up to date.


07/11/2016:  5:00PM

Cert expired on "".  Although BbLearn service was never down, user's would see the warning:  "This Connection is Untrusted", which cause numerous support center calls.  

A new cert needed to be generated by NAU and sent up to Bb Managed Hosting.  It was installed and normal service resumed on 07/12 at 10:10AM.  


06/18/2016:  1:00AM - 4:36AM

CU4 was applied on top of the October 2014 release.  

Also, the following building blocks were updated to the version listed:

Assessments v10.6.1172659
Assignment v2.6.1173798
Authentication Provider - LDAP v9.4.1171630
Blackboard Core v1.0.1172171
Caliper v1.4.1172930
Cloud Profiles v3.9.5.15121
Content Editor v11.6.1173427
Enterprise Surveys v1.3.1172553
Goal v1.5.1172931
Outcomes Assessment v1.4.1172932
Portal v1.0.1172726
SafeAssign v4.0.16
Safe HTML Filter v9.6.1170142

Finally, write.lock files were removed from /usr/local/blackboard/content/vi/BBLEARN/lucene in an effort to fix the problem of groups not deleting.
See BbSupport case #02334375 for details. 


02/27/2016:  1:01AM - 2:17AM

Database hardware resized, audit performed on nodes and the following Building Blocks were updated:

Assignment  2.6.1173155
Blackboard Mobile Web Services  94.8.0
Grading  9.10.1171491
Partner Cloud  1.6.2014101130
SafeAssign  4.0.15
WileyPLUS Integration  2.1.201410520

01/30/2016:  2:00AM - 3:20AM

LTI configuration for Norton eBook publishing was configured, followed by a rolling restart.

01/23/2016:  2:00AM - 3:10AM

Building block "Leaderboard" was installed followed by a rolling restart.

01/09/2016:  2:00AM - 5:00AM

BbLearn was upgraded to CU3 for BbLearn vOCT-2014.  Also upgraded were the following building blocks:

Assessments 10.6.1168450
Assignment 2.6.1168696
Authentication Provider - LDAP 9.4.1159866 plus patch LRNSI-18971 before installation 
Blackboard Collaborate 4.8.1172511
Blackboard Core 1.0.1170876
Blackboard Mobile Web Services 94.7.6
Blogs and Journals 1.7.1167129
Discussions 10.6.1167258
Mashup - SlideShare 18.0.1170549
Outcomes Assessment 1.4.1169180 
Partner Cloud 1.6.2014101091
Portal 1.0.1171093
Portfolios 1.3.1168443
Retention Center 1.8.1170622
SafeAssign 4.0.14
Wikis 10.6.1167160 

Note:  the "Authentication Provider" building block update reset the LDAP authentication SSL Version to "No SSL", thus when third party tools attempted to connect through LDAP, that connection failed. It was reset back to "LDAPS" on Monday (01/11), at 10:50AM to enable secure LDAP connections.

09/18/2015:  2:30PM

A new theme:  "as_2012_mod4" has been applied to BbLearn.  This theme is the same as the previous theme:  "as_2012_mod3, but has functionality to fix the "Date Picker", as described here:, KB Article No. 000038462.  A rolling restart was performed on Saturday (9/19) morning, from 2AM - 3:51AM in case node propagation was needed.

09/08/2015:  2:30PM

A new theme:  "as_2012_mod3" has been applied to BbLearn.  This theme is the same as the previous theme:  "as_2012_mod2, but has some new functionality which fixes the case where a timer appears on a users quiz/test page and cannot be moved (  Workaround #3 was applied.  A rolling restart has also been scheduled for 09/09/2015 at 2:00AM.

08/15/2015:  3:30AM - 3:49PM

BbLearn upgraded to the October 2014 release with Cumulative Update 2.  All Building Blocks were also brought up to date.  The Respondus lockdown browser was installed and the NBC Building Block was removed.

06/13/2015:  3:01AM - 3:49AM

bb-tasks.xml was updated by Managed Hosting in an effort to turn system performance.

06/05/2015:  1:00AM - 3:09AM

NAU custom building block "NAU Auto Enroll" was updated to handle more than 13 courses.  The field width (maxlength) was expanded from 256 to 512.  A rolling restart ensured that the building block was deployed to all nodes.

05/16/2015:  3:00AM - 4:15AM

Restart of production BbLearn.  The following changes were made:  1) the custom "myCourses" module was removed (BlackBoards default module takes its place); 2) The number of tools available by default was reduced to stream courses; 3) the old demo/test student building block was removed (Blackboard's "Student Preview" tool takes its place).

05/02/2015:  3:01AM - 4:09AM

Managed hosting made a change to "" from:  bbconfig.cache.QtiCanvasItemCache.needsclusterinvalidation=false

To: bbconfig.cache.QtiCanvasItemCache.needsclusterinvalidation=true

This change was made due to issues articulated in article #000039965. A rolling restart followed with each node coming back online with "pushconfigupdates"

04/11/2015:  6:47PM

Managed hosting has completed the clone from production to staging.  The production environment would not have been effected during the cloning process.

04/04/2015, 11:00PM - 04/05/2015 3:00AM

Blackboard deployed the new security patch (GHOST Vulnerability) on NAU's managed hosted production environment.  Service could have been interrupted for approximately 1 hour during this maintenance window, but it has not been ascertained if end users were affected.

04/04/2015, 1:00AM - 4:00AM

Build block MyCourseEval was disabled and this required a rolling restart in order to take effect.  End users should not have been affected.

This rolling restart had the acquired benefit of fixing NAU's BbCollaborate problems, which included users being unable to play existing recordings through BbLearn and instructors being unable to convert existing recordings to MP3/MP4.

03/21/2015, 12:00AM - 1:17AM

The achievements building block was enabled, followed by a rolling restart.  End users should not have been affected.

03/14/2015, 12:00AM - 6:51AM

During this maintenance window, the following updates were applied:

  Cumulative Update 3 (on top of the April 2014 release)

The following building blocks were updated:

  • Achievements:    1.4.1129409  (upgraded from 1.4.1095914)
  • Assessments:    10.4.1145722  (upgraded from 10.4.1121541)
  • Assignment:    2.4.1135074    (upgraded from 2.4.1096797)
  • Blackboard Mobile Web Services:    94.6.3  (upgraded from 94.5.0)
  • Cengage Learn MindLinks:  1.100.911600244   (upgraded from 1.100.911600241)
  • Course Packages - ANGEL Import:  9.4.1130631  (upgraded from 9.4.1091793)
  • Course Packages - Common Cartridge:  9.3.1122099  (upgraded from 9.3.1098070)
  • Data Integration:  1.0.1145989  (upgraded from 1.0.1095636)
  • Grading:  9.8.1128847  (upgraded from 9.8.1096075)
  • Group Manager:  1.3.1096379  (upgraded from 1.3.1135108)
  • Inline Grading:  1.3.1097417  (upgraded from 1.3.1097417)
  • My Grades:  9.51095636  (upgraded from 9.5.1135073)
  • Outcomes Assessment:  1.2.1139715  (upgraded from 1.2.1112598)
  • Partner Cloud:  1.4.201404889  (upgraded from 1.2.911600808)
  • Pearson's MyLab & Mastering:  3.2.911600568  (upgraded from 3.1.911600483)
  • Portfolios:  1.0.1142329  (upgraded from 1.0.1118955)
  • Retention Center:  1.6.1128315  (upgraded from 1.6.1096643)
  • SafeAssign:  4.0.9  (upgraded from 4.0.5)
  • Self and Peer Assessment:  16.0.1135937  (upgraded from 16.0.1096024)
  • Social Learning:  9.3.1118832  (upgraded from 9.3.1087724)
  • Software Updates:  (upgraded from

Service was restored around 4:30AM, but a rolling restart due to issues with SafeAssign extended the outtage till 6:51AM.

02/25/2015, 11:43AM - 5:40PM
Internet outage prevented users from accessing BbLearn.  Users in various geographic locations may have experienced slightly shorter or longer access problems.

01/10/2015, 2:00 - 3:16AM
Rolling restart to apply the Follett advertising module

12/13/2014, 2:00 - 3:38AM
Bb MH audit on our hosted DB.  Down time was required

12/06/2014, 2:00 - 2:59AM
Full restart of all nodes.  Service would have been affected as this was a cold restart to refresh the system ahead of finals week.

11/11/2014, 1:00 - 2:50AM
The Assessments and Content Editor building blocks were updated.  Bb Managed hosting performed a rolling restart after the updates.  Performance should not have been affected.

10/04/2014, 4:00 - 5:10AM
Bb Managed performed tuning based on results of an internal audit.  During the maintenance period, end users should not have been affected.

09/20/2014, 5:30AM
Edited theme as recommended in KB article #000037872 to fix:  "Print Preview of Discussion Forum Thread Collection Page is Cropped."  Change was successful.  No restart was necessary.

09/06/2014, 1:30AM - 4:00AM
Bb Managed Hosting applied Cumulative Update 2 to the April 2014 release.  The following building blocks were also upgraded by NAU:

  • Software Updates:  v3.8.11.14725
  • Cloud Profiles:  v3.8.9.14715
  • Social Profiles and Tools:  v3.8.9.14715
  • Partner Cloud:  v1.2.911600808
  • Bookshelf by VitalSource:  v3.1.911600017
  • Cengage Learning MindLinks:  v1.1.00.911600241
  •  v3.1.9.1160004
  • McGraw-Hill Connect and Create:  v2.2.91160026
  • Pearson's MyLab & Mastering:  v3.1.911600483
  • SafeAssign:  v4.0.5
  • Textbook:  v9.3.1124863
  • WileyPLUS:  v2.0.911600492

All academic materials messaging was also turned off per BtBb article:  "Academic Materials in Partner Cloud - Default Availability of Tools and Messaging."  Part of this process required Bb managed hosting to run 2 scripts against the DB.

Bb Managed hosting also performed some tuning to the system (BbSupport ticket #01993630).  Problems addressed included:

  • One or more stored procedures are invalid
  • Infinite loop sucks up all database resources
  • High cursor pin S wait on X waits recorded in Oracle Workload History
  • Frequent redo log switches can lead to performance issues
  • Too many sessions are trying to get exclusive locks on the SESSIONS_PK index
  • High Java utilization seen on the App and high load seen on the DB caused by Data Integration Indexer task

08/26/2014, 1:00AM - 1:32AM
BB Managed Hosting applied hotfix: LRMSI-17522, to fix user error:  "http status 508 unrecoverable error loop occurred while accessing the blackboard page."  A rolling restart was required, but this should not have affected service.

08/16/2014, 1:00AM - 9:15PM
Upgraded BbLearn to the April 2014 release.  BbLearn was offline during this upgrade.

08/06/2014, 1:00AM - 2:28AM
Pearson MyLabs/Mastering build block upgraded from v3.0.911300372 to v3.1.911300463.  Rolling restart needed to take effect.

07/26/2014, 2:03AM - 3:21AM
Installed patch LRNSI-16920 to address Apache Struts Vulnerability (CVE-2014-0114).  Rolling restart needed to take effect.

06/07/2014, 1AM - 4:30AM
Upgraded to SP13, CU10 (from CU9).  Also, the following building blocks were upgraded along with a rolling restart:

  • Assessments:  v10.1.1043226
  • Assignment:  v2.0.1053085
  • Cloud Profiles:  v3.8.5.1
  • Discussions:  v10.2.1065239
  • Grading:  v9.4.1042998
  • NBC Content Integration:  v4.0.2
  • Social Profiles and Tools:  v3.8.5.1
  • Software Updates:  v3.8.5.1

04/12/2014, 2AM - 3:56AM
The following building blocks were upgraded along with a rolling restart:

  • Collaborate:  v4.3.1066499
  • Content Editor:  v11.1.1047265
  • Group Manager:  v1.1.1025506

03/20/2014 - 03/22/2014, Blackboard managed hosting cutover
As part of the migration to Blackboard managed hosting, SP13 was upgraded to CP9

02/27/2014, 2:00AM - 6:00AM
Updated the Pearson MyLab and Mastering building block to v3.0.911300370.  Bounced all nodes.

02/20/2014, 2:00AM - 6:00AM
Upgraded the SafeAssign building block from v3.0.6 to v3.0.8
Installed the new School Chapters building block
Reinstalled the following building blocks (BB Support ticket #01881985):

  • bb-evidence
  • bb-evidence-collection
  • bb-rubric
  • bb-enterprise-survey
  • bb-goal

Edited the htmlDocument.vm and htmlDocument-start.vm (located:  ../blackboard/config/internal/renderers/page/) to fix the IE11 Security Update bug which causes BBLearn to crash when an instructor enters a course.

02/06/2014, 2:00AM - 6:00AM
Upgraded BbLearn from CP6 (SP13), to CP8.  The following building blocks were also upgraded:

  • Software Updates: -> 3.6.2
  • Cloud Profiles: -> 3.6.2
  • Social Profiles and Tools: -> 3.6.2
  • Grading:  9.4.1016900 -> 9.4.1042998
  • Inline Grading:  1.0.1038247 -> 1.0.1064516
  • Assessments:  10.1.1036575 -> 10.1.1043226
  • My Grades:  9.2.1016900 -> 9.3.1056407
  • Wikis:  10.2.1038288 -> 10.2.1064519
  • Blogs-Journals:  1.2.1037950 -> 1.2.1064522
  • Discussions:  10.2.1038288 -> 10.2.1065239
  • Self & Peer Assessment:  14.0.1037932 -> 14.0.1043247
  • Assignment:  2.0.1038248 -> 2.0.1053085
  • Content Editor:  11.1.1039380 -> 11.1.1047265
  • Collaborate:  4.2.1039053 -> 4.3.1066499
  • Retention Center:  1.3.1025511 -> 1.4.1054019
  • Admin Console:  10.1.1 -> 10.2.1030639
  • Partner Cloud:  1.0.911300486 -> 1.0.911300598
  • Calendar:  2.2.1014975 -> 2.2.1042740
  • Authentication Provider - CAS:  9.2.1011184 -> 9.3.1041563
  • Mashup - SlideShare:  14.0.1037908 -> 14.0.1065707
  • SCORM Engine:  2013.0.1016900 -> 2013.1.1042137
  • McGraw-Hill Connect and Create:  2.1.911300009 ->  2.2.91130040
  • Cengage Learning MindLinks: 1.100.911300167 -> 1.100.911300180
  • NBC Content Integration:  3.0.25 -> 4.0.1
  • Pearson MyLab / Mastering:  2.100.911300011 -> 3.0.911300369
  • Course Packages - ANGEL Import:  9.1.130093.0 -> 9.2.1032848

01/30/2014, 2:00AM - 6:00AM
Upgraded the SafeAssign building block from v2.9.2 to v 3.0.6.  This was a mandatory upgrade.  All nodes bounced.

01/12/2014, 2:00PM
Quick Setup Guide has been disabled via the GUI:  Admin panel > Course Settings > Quick Setup Guide > "Do not show the Quick Setup Guide when entering a course."

01/09/2014, 1:30PM - 4:15PM
As per BbSupport case #01869348, file org.eclipse.birt.data_2.5.2v20100209.jar has been applied to BbLearn followed by a rolling restart of all nodes.

12/19/2013, 2:00AM - 6:00AM  (weekly maintenance window)
Parameter "retain_tmp_hours" for task "bb.recyclebin.cleanup" (files: bb-tasks.xml and was changed from 48 (hours) to 24, to help alleviate the rapid buildup of files in "/bb_tmp", especially just before final exams.  All nodes were bounced.

12/11/2013, 11:59PM - 12/12/2013, 9:31AM
BbLearn was throwing errors on file upload and file access.
Directory:  /usr/local/blackboard/content/vi/BBLEARN/recyclebin/bb_tmp filled to capacity (32000 subdirectories).  It was moved to "bb_tmp_FULL" and a new one (bb_tmp) was created.  The crontab which runs "bbclean" under user "bbuser" was updated on all the nodes to run nightly.  Previously, it was set up to run once a week:  Saturday mornings at 2AM.

12/11/2013, 1:30PM
PRD8 and PRD9 were added to the BbLearn cluster

12/05/2013, 2:00AM - 6:00AM (weekly maintenance window)
Item analysis building block upgraded to v9.3.1029805; Bb mobile building block upgraded to v91.4.3  All nodes bounced.

Upgrading these 2 B2s sent the 3 trouble maker B2s (software updates, cloud profiles and social profiles and tools) out of whack.  Thus, they were upgraded as well:

  • software updates:  v3.4.11.0
  • cloud profiles:  v3.4.11.0
  • social profiles and tools:  v3.4.9

11/21/2013, 2:00AM - 6:00AM (weekly maintenance window)
The Oracle October Patch Set Update (PSU) was applied to LEARNPRD.  All nodes bounced.

11/14/2013, 3:30PM
Updated the registry_value for registry_key "portal_max_course_enrollments_to_display" in BBLEARN table "system_registry" to 1000.  This will help users who are using the default "my Courses" module who have over 100 courses to display.  The default (original) value was 100.

10/24/2013, 2:00AM - 6:00AM (weekly maintenance window)

  • All nodes upgraded to Java JDK7u45 due to security risk.
  • Xen cluster patched with Hotfix XS62E004, XS62E009 for performance and security.
  • Both dev and production EqualLogic SAN cluster upgraded to 6.0.7 to support FS7600 file system.
  • All VMs rebooted and all services bounced.  Because of the full cluster bounce, 2 changes have taken effect: 
    • file:   /usr/local/blackboard/content/vi/BBLEARN/plugins/bb-stream/webapp/WEB-INF/jsp/stream/streamSettings.jsp was edited by changing line 
      "<div class='scrollable' bb:lowestElement="streamSettingButtons" bb:dynHeightScroll="true">" to
      "<div class="scrollable scrolling" bb:lowestelement="streamSettingButtons" bb:dynheightscroll="true" style="height: 128px; overflow-y: hidden;">"
      • as recommended by BbSupport, case # 01839319"
    • file:  /blackboard/content/vi/BBLEARN/plugins/Bb-discussionboard/webapp/WEB-INF/web.xml was edited by changing line 6 from
      "version="2.5">" to "version="2.5" metadata-complete="true">" as recommended by BbSupport case # 01837987

The load balancer is no longer using round robin for session allocation and instead is using a method which determines which node has the least number of connections and allocates requests accordingly.  Using this algorithm should keep the load amongst the nodes in the pool more equally distributed.

PRD7 added to BbLearn cluster

10/09/2013, 6:30AM
B2 Content Editor (vtbe-tinymce) was upgraded to v11.1.1039380

09/26/2013, 2:00AM - 6:00AM (major upgrade)
Upgraded to SP13 / CP6 (from SP12 / CP1)

The following building blocks were also upgraded:

  • Assignment (v2.0.1038248)
  • Blogs and Journals (v1.2.1037950)
  • Cengage Learning MindLinks (v1.100.911300167)
  • Collaborate (v4.2.1039053)
  • Content Model (v1.2.1036195)
  • Discussions (v10.2.1038288)
  • Enterprise Surveys (v1.0.1019603)
  • Evidence Processing (v10.0.1019614)
  • Goal (v1.0.1019366)
  • Group Manager (v1.1.1025506)
  • Mashup - Flickr (v14.0.1037903)
  • Mashup - SlideShare (v14.0.1037908)
  • Mashup - YouTube (v14.0.1037919)
  • McGraw-Hill Connect and Create (v2.1.911300009)
  • Mobile Web Services (v91.4.1)
  • Outcomes Assessment (v1.0.1019678)
  • Partner Cloud (v1.0.911300486)
  • Pearson MyLab / Mastering (v2.100.911300011)
  • Rubrics (v1.0.1019622)
  • SafeAssign (v2.9.2)
  • Self and Peer Assessment (v14.0.1037932)
  • Track Task Progress (v1.1.1038710)
  • Wikis (v10.2.1038288)
  • WileyPLUS (v2.0.911300439)

08/21/2013, 2:00PM
bbconfig.max.permsize.tomcat now set to 768m as recommended in bbadmin-l thread "G1 Garbage Collector on SP12/13"

08/26/2013, Implemented though the day
Added the following JVM parameters to help with Full Garbage Collection. Also cleaned up order and placement.


Old lines in

New lines in

08/15/2013, 2:00AM - 3:0AM
Weekly maintenance window:  July 2013 Oracle PSU applied to LEARNPRD and run on all nodes.

06/06/2013, 5:00AM - 5:30AM
Edited:   /usr/local/blackboard/content/vi/BBLEARN/plugins/bb-stream/webapp/WEB-INF/jsp/stream/streamSettings.jsp
and changed line: <div class='scrollable' bb:lowestElement="streamSettingButtons" bb:dynHeightScroll="true">
to: <div class="scrollable scrolling" bb:lowestelement="streamSettingButtons" bb:dynheightscroll="true" style="height: 128px; overflow-y: hidden;">

Cycled all nodes with:  ""

05/30/2013, 5:00AM - 5:30AM
Added the following (all nodes) to / bbconfig.jvm.options.extra.tomcat:

  • -XX:+AggressiveOpts
  • -XX:+UseCompressedStrings
  • -XX:+UseCompressedOops
  • -XX:+OptimizeStringConcat
  • -XX:+UseStringCache

And to / bbconfig.jvm.options.gc

  • -XX:+UseParNewGC

Cycled all nodes with:  ""

05/24/2013, 2:00AM - 2:10AM
Stopped all services.  Removed /usr/loca/blackboard/content/vi/BBLEARN/activemq folder to trigger the recreation of ActiveMQ's tablespace.  Truncated table "peer_service" and then restarted BbLearn services on all nodes, one at a time.

05/23/2013, 2:00AM - 6:00AM
Upgraded to SP12, CP1.  Installed B2s:  software updates, cloud and profiles, via  Updated all other B2s which had a new version available.

04/26/2013, 2:00AM - 3:58AM
Storage is now using the new iSCSI switch.  XenServer patch XS602E013 was applied to fix NFS server issue.

04/25/2013, 2:00AM - 6:00AM
Weekly maintenance window:  The first part of a conversion to a new iSCSI switch was implemented.  All disks on the Storage Link storage repository were migrated to a standard LVM over iSCSI SR.  (This type of SR is known to work with this version of XenCenter.)  During an outage on Friday (4/27), the final cut over and re-ip of the storage array took place.

04/04/2013, 2:00AM - 6:00AM
Weekly maintenance window:  January 2013 Oracle PSU installed on LEARNPRD.  All VMs rebooted, all nodes cycled.

03/12/2013, 8:33PM - 9:02PM
Lost network connectivity to bryce and all learn nodes.

03/09/2013, 8:50PM - 9:20PM
The NFS server lost its network connection.  The network adapter had to be reset in order to restore connectivity.

03/07/2013, 5:00AM
Weekly maintenance window:  LRNSI-10005 and LRNSI-10147 applied.  All nodes cycled (./

03/05/2013, 5:00PM
admin.bblearn was taken out of the pool and is no longer public facing.

02/28/2013, 5:00AM - 6:00AM
During weekly maintenance, all the nodes were cycled (./

02/09/2013, 7:15PM - 7:54PM
The disk attached to the NFS server entered a read only state after some network issues.  This is a known issue with XenCenter and is under investigation for a long term fix.

01/30/2013, 3:00PM
TIPS (Technology: Information Proficiency Skills) turned off in BbLearn.

01/17/2013, 5:00AM - 6:00AM
During weekly maintenance, all the nodes were cycled (./

01/10/2013, 2:20AM - 6:00AM
During weekly maintenance, all the VMs were rebooted and all BbLearn nodes were restarted. was also rebooted.

01/08/2013, 7:15PM
The new theme:  "Bb Learn 2012" was applied.

01/06/2013, 12:20PM - 1:27PM
The NFS server lost connection to the disk array.  Rebooted NFS server and services are now restored.

01/02/2013, 11:30AM
The Student Support tab was added.

12/20/2012, 02:00AM - 06:00AM
During weekly maintenance window, patch LRNSI-9661 was applied:  (student's My Grades screen shows the Alignments column and the button in every row even when there are no goals set in Tools).

11/29/2012, 02:00AM - 06:00AM
During weekly maintenance window, the October PSU from Oracle was applied to LEARNPRD.

11/01/2012, 02:00AM - 06:00AM
During weekly maintenance window, patch LRNSI-9233 was applied:  (When students in a group try to access the collaboration tool in their group they get the access denied error).

10/25/2012, 02:00AM - 06:00AM
During weekly maintenance window, patch set 2 (SP9) and patch set 3 (SP9) were applied.  LRNSI-9455 was also reapplied after patch set 3, as it had been updated since its' inclusion in patch set 3.  This script was also run in LEARNPRD: rollup_backfill.sql.  What's included in patch set 2 and patch set 3 are here.

10/05/2012, 06:48AM - 08:48AM
BbLearn was unavailable due to contention on the NFS server after an upgrade to the firmware on the disks.  Several jobs on the NFS servers were stopped which restored service.

09/14/2012, 12:00PM - 12:00PM
BbLearn was unavailable due to the virtual NIC on the NFS server having lost connectivity.  No content could be retrieved which brought the system down.

09/06/2012, 9:00PM - 09/07/2012, 8:50AM
BbLearn was available for most read only tasks, but when users tried to upload files, save grades or perform other functions in BbLearn which required a write to disk, they recieved a message:  "No Space Left on Device."  The cause was Apache logs had not rotated off of 4 of the nodes:  prd1, prd3, prd4 and prd5; prd2 was fine.  Backup data was archived by hand ./logs/archives, freeing up disk space.

08/23/2012, 7:00PM - 7:50PM
BbLearn was unavailable due to the virtual NIC on the NFS server having lost connectivity.  No content could be retrieved which brought the system down.

08/23/2012, 2:00AM - 6:00AM; regularly scheduled maintenance
New batteries installed in Bryce.  Oracle July 2012 PSU applied to LEARNPRD

08/18/2012, 5:30AM
Applied patch LRNSI-8214 to address the bug:  "Thousands of "Timer-xxxxx" Being Leaked Causing Descriptor Exhaustion and Server Lock-up.

08/17/2012, 4:00PM
Added the following indexes to help with queued tasks and course copies:

  • create index COURSE_USERS_BMH_1 on COURSE_USERS(CRSMAIN_PK1,ROW_STATUS,ROLE) online tablespace BB_BB60_INDX;
  • create index EUD_ITEM_BMH_2 on EUD_ITEM (DATA_PENDING_IND,IMPORTANT_IND,CRSMAIN_PK1) online tablespace BB_BB60_INDX;

08/17/2012, 2:55PM
Set "Notification Cleanup" (System Admin -> Tools and Utilities -> Notifications) to 60 days in order to minimize notification traffic.  (was set to 120 days).

08/16/2012, 6:00AM
Upgraded to SP9, patchset 1.  Patchset 1 includes (LRN-55438, LRN-52073, LRN-51203, LRN-51725, LRN-50582, LRN-49207, LRN-46952, LRN-50259, LRN-49812, LRN-52127, LRN-48738, LRN-53498, LRN-53358, LRN-49868, LRN-51676, LRN-44772, LRN-49819, LRN-48404, LRN-54799, LRN-55374, LRN-54814)

07/06/2012, 7:40AM
admin.bblearn quit around 3:30AM (7/6/2012) and was force rebooted at 7:40AM.  Service is restored.

06/14/2012, 2:00AM
Applied April 2012 Patch Set Update (PSU) to DB LEARNPRD

05/24/2012, 1:30PM
WileyPlus building block upgraded to the latest version (2.0.91700381)

05/17/2012, 11:00AM 
Building Blocks upgraded to their latest versions:
   Bb Mobile Web Services  v91.3.1
   SafeAssign  v2.5.0
   NBC Content  v3.0.22

05/11/2012, 3:30PM
Flashback was turned back on in LEARNPRD

05/07/2012, 9:25PM - 10:05PM, BbLearn service interruption
BbLearn was down for around 40 minutes due the the EqualLogic disk array running out of available space

05/07/2012, 8:30AM
NFS recommendations from BB Support were applied (no down time was required)
   at the clients (, file system table settings were edited/added (/etc/fstab):  bg,noacl,noatime,nodiratime,rsize=32768,wsize=32768
   at the server (, in /etc/sysconfig/nfs, chaned RPCNFSDCOUNT to 32 (was set at 8)

05/05/2012, 2:00AM
BB patch LRNSI-5668 was installed.  Service was restored at 2:10AM

LRNSI-5668:  Header Injection Vulnerability in Content Collection Could Allow Tampering & High application wait and user IO since upgrade to SP6 - Causing Frequent Outages & address the full scans on XYF_FILES and XYF_URLS by hinting the getAvailableQuota() query for Oracle; also fix FileSystemUtil & Xythos DB load spikes around SUMVIEW DIGESTVIEW queries & change Quota query to be  more performant(XS-18015, XS-18016) &  Download content from WFS home with many files by XD fails and CPU usage is 100% (XS-18013). Jars are from Xythos Build: h72SP2_7_2_91_1119

05/04/2012, 7AM - 3:30PM
service was slow all day due to jam up on the NFS server.  Machines lmsvm1, lmsvm2 and lmsvm3 were rebooted at 2:00PM and service was restored at 3:30.  Changes were made systems configuration file "fstab", which now is set at: /usr/local/blackboard/content   nfs     soft,intr,rw,rsize=65536,wsize=65536,noatime,actimeo=1 0 0
Changes made include:  doubling the size of rsize and wsize, adding noatime and actimeo.

05/03/2012, 11AM - 12AM
3 more groups of redo logs (2GB each) were added to LEARNPRD

05/03/2012, 2AM - 6AM, weekly maintenance window
purge_accumulator was disabled before its nightly run (1AM)
all nodes shutdown
table "BBLEARN.ACTIVITY_ACCUMULATOR" was rebuilt using export - import; pruned down now to around 70 million rows
all nodes brought back up; purge_accumulator job started
DB standby was also turned off for a testing experiment; BB ran a stress test around 3:50AM.  Standby was turned back on at 7:26AM and was caught up by 8:16AM

05/02/2012, 4:30PM, tuning before final exam week
commented out SQL statement:  "delete from BBLEARN_stats.activity_accumulator ..." in Oracle procedure "BBLEARN_STATS.SYNC_ACTIVITY_ACCUMULATOR"
commented out DiskUsage task from /config/bb-tasks.xml and /config/
restarted admin.bblearn node for task change to take effect

05/02/2012, 8:56AM, prd5 restart
prd5 did not come up cleanly in the cluster restart earlier this morn (errors in bb-services-log.txt:  "Could not initialize class ...") and had to be restarted

05/02/2012, 2:15AM, cluster restart
full cluster was restart to see how this would impact stress testing through the load balancer

04/27/2012, 1:00AM, purge_accumulator now running with a new "days to keep" argument
"days to keep" changed from 180 to 90 for the purge_accumulator job

04/26/2012, 9:40AM, Network connectivity restored VM had been in some funky network state (we've seen this before with other nodes), and the NFS service had to be restarted.

04/24/2012, 02:00AM, Flashback turned off and patch applied
flashback was turned off in LEARNPRD (since 4PM, 04/23)

BB patch LRNSI-7129 was applied to help alleviate problems with long running query against tables "xyf_files" and "xyf_urls"
   LRNSI-7129:  Prevent 360 degree view report from listing children for top level directories
     Resolved bugs:  LRN-51675

04/18/2012, 02:00PM, BB Support suggested maintenance
backed up table CMS_FILE_ACCESS and truncated it (27,976,468 rows)

Also, noted that 2 images: and iOS-Blackberry.png were still being loaded out the of content system.  They are now being served from the nodes.

04/10/2012, 02:00AM, cluster restart
BBLearn cluster was restarted (PushConfigUpdates) for changes made in the afternoon of 04/09/12 (maxclients 500 -> 200) to take effect.

04/10/2012, 12:06AM, VM rebooted VM lost network connectivity which required a reset of the NIC via Xen and a restart.  Service was restored at 12:12AM

04/09/2012, 4:30PM, changes made to PRD environment as recommended by BB support/audit (Stephen Byrne)

JDBC settings for CMS document store (via GUI) were changed from:
   Maximum Connections:  25 -> 100
   Minimum Connections:    2 -> 25

added to /etc/sysctl.conf: 
   net.ipv4.tcp_keepalive_time = 1800
   fs.file-max = 65536

added to /etc/secutiry/limits.conf:
  *               soft     nofile          65536
  *               hard     nofile          65536

added to /etc/bashrc:
  ulimit -Hn 65536
  ulimit -n  65536

changed bbconfig.appserver.max.spare.threads, bbconfig.appserver.maxthreads and bbconfig.unix.httpd.maxclients in from 500 to 200

changed MaxClients in /usr/local/blackboard/apps/httpd/conf/httpd.conf from 500 to 200

04/05/2012, 8:45AM, VM rebooted VM had been in some funky state since 12:48AM and had to be force rebooted.  prd2 was back up and BBLearn available on at 8:47AM.  End users were not affected.

04/04/2012, 3:14PM, admin.bblearn started and then PushConfigUpdates (3:27PM)
admin.bblearn was not running, and was started.  Perhaps it never came back up from PushConfigUpates this morn?  But, a simple start would not get the exec service running.  Finally did a and admin is now responsive.

04/04/2012, 9:05AM, bounced
Huge DB spike (8:49AM - 9:11AM) exhausted all available sessions (2500) in the LEARNPRD DB. was bounced (PushConfigUpdates) in order to stop ongoing processing in hopes of reclaiming DB sessions.  One more spike occurred (9:25AM - 9:38AM). All quiet since.

There were also 2 "normal" (Abbie-Normal? (wink)) spikes earlier in the morn:  4:48 - 4:56AM and 5:24 - 5:31AM.

04/03/2012, 6:10PM, bounced was bounced (PushConfigUpdates) so purge accumulator would only run once (change had been made to bb-tasks.unix.xml; see previous log entry).  Morning of 4/4 showed only 1 instance of Purge Accumulator running, so all's good.

04/03/2012, 3:10PM, Changes to bb-tasks.unix.xml and
Last Thursday (03/29/2012), when nodes were bounced, admin node "bb-tasks.unix.xml" reverted back to running both Purge Accumulator and Rotate logs because the update (commenting out of jobs) had not been made to ""  Both files have now been changed (both jobs commented out in xml files) and admin.bblearn will be bounced soon so changes will take effect before tonight.

04/03/2012, 10:30AM, Full Text Indexing time limit
Set a 60 minute time limit in "" in "/home/bbuser/fullTextIndexing" to prevent multiple runs from occurring.  Previously, this job had no time limit.

04/03/2012, 9:38AM, Probe Change for Load Balancer
The health check probe for the load balancer was changed from /index.html to /webapps/portal/healthCheck and expected status changed to 200 299

04/03/2012, 9:38AM, bounced was bounced to clean it out and relieve burden on DB.  Memory on was increased to 18GB.  BB consultant was having trouble running snapshot scripts to create fake courses for load testing. was back up at 9:44AM.

03/30/2012, 2:00PM, Login page images moved
The custom images which load on the login page were moved out of the content area and into /usr/local/blackboard/content/vi/BBLEARN/branding/login (on all nodes) where they can be served by apache.  This was recommended by BB Support as part of the DB Spike ticket.

03/29/2012, 4:29PM, lookup_link_target_icon_url patch applied
A recursive link was created in a class that caused DB performance issues. This is a known issue (LRNSI-2710).  Bb support provided an updated procedure for "lookup_link_target_icon_url." All sessions stuck using the old version of the procedure were killed so the new procedure could be put into place. The system was slow, but never became unresponsive.  Other than users being unable to access the class in question (1121-5894), end users should not have been effected.

03/29/2012, 1:26PM, Admin node bounced
The admin node was bounced in hopes of clearing a heavy DB load.

03/29/2012, 5:45AM, Weekly maintenance window restart
All nodes restarted (  The admin node was back up at 5:20AM.

03/28/2012, 10:25AM, WileyPlus Building Block upgrade
The WileyPlus Building Block was upgraded to v1.1

03/26/2012, 3:15PM, restart admin
Restarted admin because of a weird network disconnect from Xen.

03/26/2012, 1:00PM, prd5.bblearn restart
Prd5 had crashed on 3/25 at 11:30AM and was rebooted to be brought back into service.

03/26/2012,  11:30AM, setting changes

  • in bb-tasks.unix.xml, commented out both "bb.stats.purging" (purge accumulator) and bb.log.rotation" since both are controlled on admin.bblearn by crontab
  • in, set both "bbconfig.appserver.max.spare.threads" and "bbconfig.appserver.maxthreads" to 500, to conform to "bbconfig.unix.httpd.maxclients" (Apache)
  • in bb-tasks.xml, changed the "delay" parameter in the "DomainCollectionTask" from "<property name="delay" value="1000" />" to "<property name="delay" value="21600000" />"

03/26/2012,  9:30AM, setting change
Changed System Admin -> Feature and Tool Management > Feature and Tool Availability > Statistics Available to "No" as advised in BB Case ID:  935025 regarding table "CMS_FILE_ACCESS"

03/24/2012,  1:17PM, Admin Restart.
Learn was running slow due to many UPDATE sessions SET timestamp = :1 WHERE session_id = :2 running. Restarted Admin server to see if it would help this issue. No end users should have been kicked off.

03/23/2012  4:56PM, Admin node restart
The admin node was restarted as it was doing full gc.  End users should not have been affected.

03/15/2012  4:00 AM:  Weekly maintenance window
LEARNPRD was upgraded to Oracle  All nodes were started back up with PushConfigUpdates.shThe admin node was back up at 3:37AM.

03/14/2012  7:45 AM:  admin node bounced
The BbLearn admin node was bounced (shutdown followed by "") to see how it would effect the LEARNPRD spikes.  End users were not effected.

03/09/2012  12:15 PM:  Truncating Sessions and sessions_inds and rebuilding indexes
DB spikes since 5:40AM were hindering system performance.

All BbLearn nodes were shutdown.
The following DDL was executed:

 Push config was executed to bring the servers back up.

03/08/2012  5:30 AM:  Weekly maintenance window
All BbLearn nodes were cycled (./ thus invoking new setting in config/tomcat/classes/ of EventBasedIndexingEnabled= false.

03/07/2012  09:55 AM:  change to time to rotate logs and indexing param change
Changed time to rotate logs to 1:00AM because the McGraw Hill B2 was rotating its log at the same old midnight time and was causing the delivered Bb script to fail.
Update config/tomcat/classes/[.bb] and set EventBasedIndexingEnabled=false. This will not take place until the restart planed for the morning. 

03/07/2012  09:55 AM:  change to run time of is run daily to create a folder for every course so the library folks have a place to put library reserve content.  This job runs about 2 hours and was (pre 3/7) scheduled to run at 11PM (via crontab, user bbuser, admin node).  It is now set to run at 2AM.

03/06/2012  10:15 AM:  full cluster restart
BbLearn was unresponsive.  Did a full cluster restart.  This error was in bb-services-log.txt: "Failed to run cache update. - java.lang.NullPointerException" on all nodes, after the full cluster restart.  Ran on prd1.bblearn which seems to have cleared up the problem.

03/02/2012  4:00 PM:  admin node bounced
The BbLearn admin node was bounced to see how it would effect the LEARNPRD spikes we've been seeing since we bounced the cluster yesterday morn.  End users were not effected.

03/01/2012  5:30 AM:  Weekly maintenance window
All BbLearn nodes were cycled (./ and patch LRNSI-6358 was applied.

02/15/2012  10:00AM - 11:05AM:  BbLearn slowdown/unavailable
BbLearn was slow and unresponsive during the morning, which could have been partially due to a full backup running.  All nodes and LEARNPRD were cycled.  SGA memory was increased from 8GB to 12G and the PGA memory was increased from 4GB to 6GB.  Ticket was opened with BbSupport and AWR report was enclosed.

02/09/2012 06:00AM Normal LMS maintenance window.
LEARNPRD temp tablespace was resized to 16GB.  The number of database processes was increased from 1600 to 2500 in light of 2 new nodes (prd4.bblearn and prd5.bblearn) being added.

02/05/2012 11:00AM - 1:42PM; BbLearn was down/unresponsive due to LEARNPRD being under heavy load
All nodes and LEARNPRD were restarted due to heavy DB load.  TEMP table space was filled.  Bb Patch LRNSI-5672 was applied (latest Xythos patch).  This seems to be a new problem, as the DB blocking lockers are different from our last DB incident.

02/03/2012 02:45PM PRD3 Restarted
PRD3 was restarted as it was doing full garbage collection.

02/03/2012 11:30AM PRD3 Restarted
All nodes were swamped, as load was heavy.  LEARNPRD was OK.  PRD3 was restarted to bring it back from beyond.

2 new nodes were added to the cluster, prd4.bblearn and prd5.bblearn.

01/26/2012 06:00AM Normal LMS maintenance window.
Bb patch LRNSI-5808.bbp applied.

Added to /apps/httpd/conf/ and /apps/httpd/conf/ssl.conf:
   SSLCertificateChainFile /usr/local/blackboard/content/ssl/nau.chain

Updated /config/

01/25/2012 01:07AM - 2:30PM BbLearn performance sporadic; Production nodes and LEARNPRD bounced.
LEARNPRD bounced due to huge number of blocking lockers.  During down time, tempspace was resized from 32GB to 30GB

01/24/2012 10:56PM Admin node bounced.
Admin node doing full garbage collection and was restarted.  This should not have effected users.

01/23/2012 07:20AM - 7:50AM Production nodes and LEARNPRD bounced.
LEARNPRD bounced due to huge CPU spike on bryce as well as sustained I/O waits and blocking lockers

01/19/2012 06:00AM Normal LMS maintenance window.
mod_gzip turned off on all Apache web servers across all nodes and Apache restarted.

01/17/2012 08:50PM BbLearn not responding.
Apache died across all nodes.  Apache was restarted and admin.bblearn was added to the pool which now includes 4 nodes  Service was restored by 9:20PM.

01/12/2012 07:00AM WileyPLUS/Blackboard Integration Building Block Installed on Bb Learn.
The building block to integrate WileyPLUS with Bb Learn was installed and configured.

01/12/2012 06:00AM Upgrade to Bb Learn 9.1 Sp7 HF1 Patch Set 1
Learn was upgraded to the latest version. This included a Service Pack, a Hot Fix and a Patch Set. As of now the following patches are installed:

LRNSI-5950  2012-01-12 03:13:14 -0700   2011-12-14 14:09:20 -0700   LRN-39265  
LRNSI-5955  2012-01-12 03:12:33 -0700   2011-12-14 14:23:14 -0700   LRN-28887  
LRNSI-5952  2012-01-12 03:13:28 -0700   2011-12-14 14:13:24 -0700   LRN-41624  
LRNSI-5946  2012-01-12 03:13:48 -0700   2011-12-14 13:58:45 -0700   LRN-43622  
LRNSI-5959  2012-01-12 03:11:53 -0700   2011-12-14 16:18:02 -0700   LRN-31984  
LRNSI-5957  2012-01-12 03:12:08 -0700   2011-12-14 16:31:51 -0700   LRN-30399 
LRNSI-5951  2012-01-12 03:12:46 -0700   2011-12-16 11:34:43 -0700   LRN-28742, LRN-33359
LRNSI-5961  2012-01-12 03:11:36 -0700   2011-12-14 16:21:33 -0700   LRN-44451  

During the maintenance window, the following changes were made on bryce:

1. Increased the size of /dblog1 and /dblog2 from 10GB to 50GB

2. Made Kernel changes so dirty pages in memory are written out more frequently:

  • vm.dirty_background_ratio = 3
  • vm.dirty_ratio = 15
  • vm.dirty_expire_centisecs = 500
  • vm.dirty_writeback_centisecs = 100

3. Changed the kernel parameter that determines how aggressive swapping is done.  Set to 0 so swapping is only done when needed.

  • vm.swappiness = 0

4. Configured hugepages for Oracle which makes the memory pagesize larger for Oracle processes (this memory is not swappable)

  • vm.nr_hugepages = 6656
  • vm.hugetlb_shm_group = 101
  • ulimit settings for Oracle:
  • oracle soft memlock 13631488
  • oracle hard memlock 13631488

The Following changes were made to LEARNPRD:

  1. set SGA_TARGET to 8G
  3. reset MEMORY_TARGET
  4. set USE_LARGE_PAGES to “only” (ensures that the allotted space is available on database startup)

01/10/2012 08:20AM chksum turned off in LEARNPRD
The db_block_checking parameter was set to "FALSE" (was previously set to "TRUE").

01/10/2012 07:25AM PRD3 Restarted
PRD3 was restarted in an attempt to clear grade book long sessions (UPDATE GRADEBOOK_MAIN SET ...).

12/22/2011 05:00AM Scheduled Maintenance
Apply Oct2011 Oracle PSU to LEARNPRD.bryce and LEARNPRD.chaco

12/15/2011 10:05PM Added another node
Added PRD3 back into the cluster.

12/14/2011 12:30PM Full Restart
BbLearn, LearnPRD and bryce were bounced alleviate heavy load on LEARNPRD (bryce).  BbLearn cluster was brought up on only 2 nodes, PRD1 and PRD2.  Service was restored by 1:25PM.

12/14/2011 11:10AM Full Restart
BbLearn and LearnPRD were bounced alleviate heavy load on LEARNPRD (bryce).  Service was restored by 11:38AM.

12/14/2011 7:30AM Full Restart
BbLearn was bounced alleviate heavy load on LEARNPRD (bryce).  Service was restored by 8:05AM.

12/13/2011 1:45PM Full Restart
BbLearn was bounced alleviate heavy load on LEARNPRD (bryce).  admin node was added as a fourth node to the cluster.  Service was restored by 2:15PM.

12/12/2011 6:30PM Full Restart
BbLearn and LEARNPRD were shut down and restarted to alleviate an indeterminate load on LEARNPRD (bryce).  Service was restored by 7:55PM. 

12/11/2011 8:30PM Full Restart
After a rolling restart, BbLearn was shut down and restarted to alleviate an indeterminate load on LEARNPRD (bryce).  Service was restored by 8:45PM.  Later (around 11PM), service again became sluggish.  LEARNPRD had all of its 32GB of temp space full.  BbLearn became stable again around 11:30 PM (perhaps after enough users gave up?).

12/04/2011 7:30PM Rolling restart and Full Restart
After a rolling restart, BbLearn was shut down and restarted to alleviate an indeterminate load on LEARNPRD (bryce).  Service was restored by 7:45PM.

11/21/2011 10:30PM Rolling restart
Due to heavy load, service was unavailable or inconsistent.  Error messages varied; some pages loaded, others loaded partially, some not at all.  A rolling restart was performed on all nodes. Service was restored by 11:25PM.

11/07/2011 10:30PM Full Cluster restart
Problems with prd3 forced the other two nodes to take too many connections forcing a full restart. Service was restored by 10:50PM.

11/03/2011 02:50PM PRD1 disk at 100%
prd1.bblearn's drive mount partition which holds content (logs) reached 100%.  Users connecting to prd2 and prd3 were not affected.  Users on prd1 attempting to upload content did recieve an error message.  Log files were off-loaded to alleviate the problem.  Full service was restored by 3:25 PM.

10/31/2011 07:50AM PRD2 restarted
prd2.bblearn had been unreachable since 7:46 PM, Sunday 10/29, and was restarted.

10/20/2011 02:00AM Scheduled Maintenance
All Bb Learn application servers were restarted.   

10/06/2011 02:00AM Scheduled Maintenance
rollback LRNSI-4851(Original Quiz Access Denied Patch) , apply 886515C8R1LRNSI-4851_v4.bbp(Quiz Access Denied Patch), apply LRNSI-5098.bbp(Fix PDF in OSX)

09/22/2011 02:00AM Scheduled Maintenance
All nodes including NFS, admin and prd1-3 were restarted. New Xythos patch applied via ./ apply 347146_LRNSI-4148.bbp.

09/19/2011 10:40AM NFS Parameter change
The noatime parameter was set for the learnContent mount point to improve NFS performance.

09/15/2011 2:00AM-6:00AM Scheduled Maintenance

  • BbPatch was upgraded to 2.0 on all app nodes
  • Patch AS-162528 was applied to fix pdf rendering issues within learning modules
  • Patch LRNSI-4851 was applied to fix accedd denied errors that were showing up while taking quizzes

09/13/2011 10:00AM Performance Tuning

  • cursor_sharing was set back to exact

09/01/2011 3:00PM Performance Tuning
Based on recommendations from Blackboard (ticket #882730) some parameters were adjusted to improve performance.

  • analyze_my.bbtabs and dbms_utility.analyze_schema('SYS','COMPUTE'); db jobs were set to broken
  • The NotificationRemoveStaleDataTask was set to run once per day
  • The DistributionSendNotificationTask and DistributionSendNotificationTask were set to run once per hour
  • cursor_sharing was set to similar

09/01/2011 2:00AM-6:00AM Scheduled Maintenance
JVM tuning and patches applied. Each node was bumped up to 8 VCPU's. The heap and permgen were increased from 10GB, 512M to 12GB,1024M respectfully. and  The ¿XX:ParallelCMSThreads and XX:ParallelGCThreads parameters were bumped from 2, 4 to 3, 8 respectfully. A one-off Xythos patch was applied along with patch AS-162462.bbp that resolves AS-162460, Connection leak in CourseEnrollmentManagerImpl$CheckDomainUserQuery due to incorrect use of JdbcQueryHelper.

08/25/2011 2:00AM-6:00AM Scheduled Maintenance
The Learn database server had memory added to it and the MEMORY_MAX_TARGET was bumped to 12GB.

08/04/2011 2:00AM-6:00AM Scheduled Maintenance
All nodes were bumped up from 2 vcpus to 4 vcpus. An updated my courses module was installed. Timeout values were fixed.

07/28/2011 2:00AM-6:00AM Scheduled Maintenance
Bb Learn was upgraded to service pack 6

06/30/2011 2:00AM Scheduled Maintenance -- Disk Resize, Timeouts and Memory
All nodes have had the blackboard install moved to its own partition. All timeouts where changed to 120 mins. Admin had its memory increased. 

06/24/2011 6:05AM prd1 restart
prd1 was restarted because it was using 100% of its CPU for several days. This should not have affected end users.

06/23/2011 2:00AM-6:00AM Scheduled Maintenance
The main tomcat session timeout was changed fromt the default of 30 minutes to 120 minutes on prd2. Also, the stack memory was changed from 320Kb to 1024Kb on prd2. This was done on prd2 only to guage the impact of memory usage and performance of making this change across the cluster. prd1 and prd2 were then restarted.

06/20/2011 7:20AM prd2 restart
prd2 was restarted because it was using 100% of its CPU for several days. This should not have affected end users.

06/16/2011 2:00AM-6:00AM Scheduled Maintenance
Applied Blackboard Learn patch BUG:AS-158601 PATCH:AS-160257 - Cross-site Scripting Vulnerability in Blackboard Learn Discussion Board Could Allow Elevation of Privilege and Xen Server hotfix XS56E006 to the Xen cluster.

06/09/2011 2:00AM-6:00AM Scheduled Maintenance
Full text indexing was disabled from the GUI and immediate indexing was disabled for Xythos as per Bb Support's instructions. Full text indexing was then scheduled to run via command line script through cron.

06/06/2011 12:20AM prd2 restart
prd2 was restarted because it was using 100% of its CPU for several days. This should not have affected end users.

06/06/2011 12:15AM prd1 restart
prd1 was restarted because it was using 100% of its CPU for several days. This should not have affected end users.

06/02/2011 5:00PM-6:15PM SSL Outage
The SSL certificate that was hosted on the nodes expired. This caused the ACE to stop trusting the nodes. The cert was updated on the ACE, but since the ACE plays man-in-the-middle it still needed to have a valid, non-expired cert on the nodes. The cert was updated and the nodes restarted.

06/02/2011 2:00AM Upgrade to SP5
BBLearn was upgraded to SP5 on all nodes. Patches BUG:AS-154719 PATCH:AS-159089.bbp, BUG:AS-155049 PATCH:AS-159092.bbp and BUG:AS-155143 PATCH:AS-159210 applied. Memory on and upgraded to 14Gig from 12Gig. Theme also updated to remove blue arrows.

05/26/2011 2:00AM New SSL Cert and Oracle CPU
Oracle was updated to the latest CPU. A new SSL cert was installed. This new cert is using the NAU global sign CA. 

04/25/2011 12:00PM LMSfeed 1.7.1 deployed
A new version of the integration feed was deployed that should speed up operations.

04/11/2011 2:00AM - 6:00AM Scheduled Maintenance
The memory_max_target on the production database for LEARPRD was decreased by 1GB to alleviate the excessive swapping that was happening on Bryce. This should improve performance.

04/04/2011 12:00PM LMSfeed 1.7 deployed
A new version of the integration feed was deployed. This new feed allows for combined sections in Bb Learn.

03/24/2011 - Scheduled Maintenance - Cluster Reboot
The cluster was restarted to allow the new kernel updates to take effect.

03/18/2011 9:00AM - Restart and memory increase
Both production VM's had their available memory increased to 12GB each and then restarted in turn. This should not have affected end users.

03/14/2011 10:25AM - BBLEARN_STATS cchema altered
The following sql was executed to fix a schema mismatch that was causing the PurgeAccumulator job to fail.

03/03/2011 - Scheduled Maintenance - Cluster Reboot
The cluster was restarted to allow the new kernel updates to take effect. 

02/28/2011 - RedHat Patches Applied
No downtime. All updates from RHN have been applied to the Learn cluster. New kernel will be used next reboot.

02/17/2011 1:30PM - Disables eReserves tool
The eReserves tool is dysfunctional to the point that we are not going to use it, so the tool was disabled from the content collection.

02/17/2011 8:35AM - New NAU theme made default
A new NAU theme was uploaded and made the system default. The only change to this theme from the Bb Learn theme at this point is that the course title listed in the top of the course menu is now hidden.

02/17/2011 8:30AM - Modified custom login page
Small change was made to custom login page to change the link to e-Learning from to so that it would fit in the page column width. New page was uploaded to prd1 and prd2.

02/17/2011 2:00AM-6:00AM - Scheduled Maintenance
Oracle upgraded to and January CPU applied.

02/15/2011 8:58AM - pinned my courses module
The MyCourses module was removed from the default tab layout and then added back by pinning it to the top center column. All users should now see the MyCourses module at the top of the center column on the Home tab and now cannot minimize or move that module.

02/13/2011 2:04PM-2:17PM - bblearn down due to database crash
The database that is used by crashed which caused bblearn to be inaccessible for a time. The database was restarted and Bb Learn came back online.

02/10/2011 10:05PM-10:50PM - bblearn down due to database crash
The database that is used by crashed which caused bblearn to be inaccessible for a time. The database was restarted and checked for proper health before restarting the bblearn service.

02/10/2011 2:00AM-6:00AM - Scheduled Maintenance

  • Blackboard patch AS-155324 was applied to fix a quiz submission issue
  • The system-wide session timeout was changed from 3 hours to 2 hours
  • The assessments tool timeout was changed from 20 minutes to 2 hours
  • System tasks were disabled on to help improve overall performance and stability

01/21/2011 3:40PM - prd1 restart and permgen settings change ran out of PermGen space and needed to be restarted. This should not have affected end users. The permgen setting was changed from 256MB to 512MB and the pushupdates script was run so that it will take effect.

01/27/2011 2:00AM-6:00AM - Scheduled Maintenance
The oracle parameter, FILESYSTEMIO_OPTIONS=SETALL was remove from Bryce for LEARDPRD and VISTA4. LEARNPRD and Vista4 were then restarted.

01/21/2011 2:30PM - prd1 restart ran out of PermGen space and needed to be restarted. This should not have affected end users.

01/18/2011 11:50AM - prd2 restart and permgen settings change ran out of PermGen space and needed to be restarted. This should not have affected end users. The permgen setting was also changed from 256MB to 512MB to try to keep this from happening again. was also changed from 256MB to 512MB which will take effect the next time it is restarted.

01/12/2011 7:00AM - 8:00 AM - Upgrade
Learn was upgraded to (9.1 sp4). Prd2 was also configured but not yet added to the server farm.

01/03/2011 8:00AM - 5:00 PM - Scheduled Maintenance
The new installation (9.1 sp3) and database were brought online as

  • No labels