Skip to end of metadata
Go to start of metadata

06/07/2014, 1AM - 4:30AM
Upgraded to SP13, CU10 (from CU9).  Also, the following building blocks were upgraded along with a rolling restart:

  • Assessments:  v10.1.1043226
  • Assignment:  v2.0.1053085
  • Cloud Profiles:  v3.8.5.1
  • Discussions:  v10.2.1065239
  • Grading:  v9.4.1042998
  • NBC Content Integration:  v4.0.2
  • Social Profiles and Tools:  v3.8.5.1
  • Software Updates:  v3.8.5.1

04/12/2014, 2AM - 3:56AM
The following building blocks were upgraded along with a rolling restart:

  • Collaborate:  v4.3.1066499
  • Content Editor:  v11.1.1047265
  • Group Manager:  v1.1.1025506

03/20/2014 - 03/22/2014, Blackboard managed hosting cutover
As part of the migration to Blackboard managed hosting, SP13 was upgraded to CP9

02/27/2014, 2:00AM - 6:00AM
Updated the Pearson MyLab and Mastering building block to v3.0.911300370.  Bounced all nodes.

02/20/2014, 2:00AM - 6:00AM
Upgraded the SafeAssign building block from v3.0.6 to v3.0.8
Installed the new School Chapters building block
Reinstalled the following building blocks (BB Support ticket #01881985):

  • bb-evidence
  • bb-evidence-collection
  • bb-rubric
  • bb-enterprise-survey
  • bb-goal

Edited the htmlDocument.vm and htmlDocument-start.vm (located:  ../blackboard/config/internal/renderers/page/) to fix the IE11 Security Update bug which causes BBLearn to crash when an instructor enters a course.

02/06/2014, 2:00AM - 6:00AM
Upgraded BbLearn from CP6 (SP13), to CP8.  The following building blocks were also upgraded:

  • Software Updates:  3.4.11.0 -> 3.6.2
  • Cloud Profiles:    3.4.11.0 -> 3.6.2
  • Social Profiles and Tools:  3.4.9.0 -> 3.6.2
  • Grading:  9.4.1016900 -> 9.4.1042998
  • Inline Grading:  1.0.1038247 -> 1.0.1064516
  • Assessments:  10.1.1036575 -> 10.1.1043226
  • My Grades:  9.2.1016900 -> 9.3.1056407
  • Wikis:  10.2.1038288 -> 10.2.1064519
  • Blogs-Journals:  1.2.1037950 -> 1.2.1064522
  • Discussions:  10.2.1038288 -> 10.2.1065239
  • Self & Peer Assessment:  14.0.1037932 -> 14.0.1043247
  • Assignment:  2.0.1038248 -> 2.0.1053085
  • Content Editor:  11.1.1039380 -> 11.1.1047265
  • Collaborate:  4.2.1039053 -> 4.3.1066499
  • Retention Center:  1.3.1025511 -> 1.4.1054019
  • Admin Console:  10.1.1 -> 10.2.1030639
  • Partner Cloud:  1.0.911300486 -> 1.0.911300598
  • Calendar:  2.2.1014975 -> 2.2.1042740
  • Authentication Provider - CAS:  9.2.1011184 -> 9.3.1041563
  • Mashup - SlideShare:  14.0.1037908 -> 14.0.1065707
  • SCORM Engine:  2013.0.1016900 -> 2013.1.1042137
  • McGraw-Hill Connect and Create:  2.1.911300009 ->  2.2.91130040
  • Cengage Learning MindLinks: 1.100.911300167 -> 1.100.911300180
  • NBC Content Integration:  3.0.25 -> 4.0.1
  • Pearson MyLab / Mastering:  2.100.911300011 -> 3.0.911300369
  • Course Packages - ANGEL Import:  9.1.130093.0 -> 9.2.1032848

01/30/2014, 2:00AM - 6:00AM
Upgraded the SafeAssign building block from v2.9.2 to v 3.0.6.  This was a mandatory upgrade.  All nodes bounced.

01/12/2014, 2:00PM
Quick Setup Guide has been disabled via the GUI:  Admin panel > Course Settings > Quick Setup Guide > "Do not show the Quick Setup Guide when entering a course."

01/09/2014, 1:30PM - 4:15PM
As per BbSupport case #01869348, file org.eclipse.birt.data_2.5.2v20100209.jar has been applied to BbLearn followed by a rolling restart of all nodes.

12/19/2013, 2:00AM - 6:00AM  (weekly maintenance window)
Parameter "retain_tmp_hours" for task "bb.recyclebin.cleanup" (files: bb-tasks.xml and bb-tasks.xml.bb) was changed from 48 (hours) to 24, to help alleviate the rapid buildup of files in "/bb_tmp", especially just before final exams.  All nodes were bounced.

12/11/2013, 11:59PM - 12/12/2013, 9:31AM
BbLearn was throwing errors on file upload and file access.
Directory:  /usr/local/blackboard/content/vi/BBLEARN/recyclebin/bb_tmp filled to capacity (32000 subdirectories).  It was moved to "bb_tmp_FULL" and a new one (bb_tmp) was created.  The crontab which runs "bbclean" under user "bbuser" was updated on all the nodes to run nightly.  Previously, it was set up to run once a week:  Saturday mornings at 2AM.

12/11/2013, 1:30PM
PRD8 and PRD9 were added to the BbLearn cluster

12/05/2013, 2:00AM - 6:00AM (weekly maintenance window)
Item analysis building block upgraded to v9.3.1029805; Bb mobile building block upgraded to v91.4.3  All nodes bounced.

Upgrading these 2 B2s sent the 3 trouble maker B2s (software updates, cloud profiles and social profiles and tools) out of whack.  Thus, they were upgraded as well:

  • software updates:  v3.4.11.0
  • cloud profiles:  v3.4.11.0
  • social profiles and tools:  v3.4.9

11/21/2013, 2:00AM - 6:00AM (weekly maintenance window)
The Oracle October Patch Set Update (PSU) was applied to LEARNPRD.  All nodes bounced.

11/14/2013, 3:30PM
Updated the registry_value for registry_key "portal_max_course_enrollments_to_display" in BBLEARN table "system_registry" to 1000.  This will help users who are using the default "my Courses" module who have over 100 courses to display.  The default (original) value was 100.

10/24/2013, 2:00AM - 6:00AM (weekly maintenance window)

  • All nodes upgraded to Java JDK7u45 due to security risk.
  • Xen cluster patched with Hotfix XS62E004, XS62E009 for performance and security.
  • Both dev and production EqualLogic SAN cluster upgraded to 6.0.7 to support FS7600 file system.
  • All VMs rebooted and all services bounced.  Because of the full cluster bounce, 2 changes have taken effect: 
    • file:   /usr/local/blackboard/content/vi/BBLEARN/plugins/bb-stream/webapp/WEB-INF/jsp/stream/streamSettings.jsp was edited by changing line 
      "<div class='scrollable' bb:lowestElement="streamSettingButtons" bb:dynHeightScroll="true">" to
      "<div class="scrollable scrolling" bb:lowestelement="streamSettingButtons" bb:dynheightscroll="true" style="height: 128px; overflow-y: hidden;">"
      • as recommended by BbSupport, case # 01839319"
    • file:  /blackboard/content/vi/BBLEARN/plugins/Bb-discussionboard/webapp/WEB-INF/web.xml was edited by changing line 6 from
      "version="2.5">" to "version="2.5" metadata-complete="true">" as recommended by BbSupport case # 01837987

10/23/2013
The load balancer is no longer using round robin for session allocation and instead is using a method which determines which node has the least number of connections and allocates requests accordingly.  Using this algorithm should keep the load amongst the nodes in the pool more equally distributed.

10/22/2013
PRD7 added to BbLearn cluster

10/09/2013, 6:30AM
B2 Content Editor (vtbe-tinymce) was upgraded to v11.1.1039380

09/26/2013, 2:00AM - 6:00AM (major upgrade)
Upgraded to SP13 / CP6 (from SP12 / CP1)

The following building blocks were also upgraded:

  • Assignment (v2.0.1038248)
  • Blogs and Journals (v1.2.1037950)
  • Cengage Learning MindLinks (v1.100.911300167)
  • Collaborate (v4.2.1039053)
  • Content Model (v1.2.1036195)
  • Discussions (v10.2.1038288)
  • Enterprise Surveys (v1.0.1019603)
  • Evidence Processing (v10.0.1019614)
  • Goal (v1.0.1019366)
  • Group Manager (v1.1.1025506)
  • Mashup - Flickr (v14.0.1037903)
  • Mashup - SlideShare (v14.0.1037908)
  • Mashup - YouTube (v14.0.1037919)
  • McGraw-Hill Connect and Create (v2.1.911300009)
  • Mobile Web Services (v91.4.1)
  • Outcomes Assessment (v1.0.1019678)
  • Partner Cloud (v1.0.911300486)
  • Pearson MyLab / Mastering (v2.100.911300011)
  • Rubrics (v1.0.1019622)
  • SafeAssign (v2.9.2)
  • Self and Peer Assessment (v14.0.1037932)
  • Track Task Progress (v1.1.1038710)
  • Wikis (v10.2.1038288)
  • WileyPLUS (v2.0.911300439)

08/21/2013, 2:00PM
bbconfig.max.permsize.tomcat now set to 768m as recommended in bbadmin-l thread "G1 Garbage Collector on SP12/13"

08/26/2013, Implemented though the day
Added the following JVM parameters to help with Full Garbage Collection. Also cleaned up order and placement.

-XX:+AggressiveOpts 
-XX:+AlwaysPreTouch 
-XX:CMSFullGCsBeforeCompaction=1
-XX:+CMSParallelRemarkEnabled
-XX:+OptimizeStringConcat
-XX:+UseCompressedOops 
-XX:+UseCompressedStrings 
-XX:-UseSplitVerifier 
-XX:+UseStringCache 

Old lines in bb-config.properties

New lines in bb-config.properties

08/15/2013, 2:00AM - 3:0AM
Weekly maintenance window:  July 2013 Oracle PSU applied to LEARNPRD and PushConfigUpdates.sh run on all nodes.

06/06/2013, 5:00AM - 5:30AM
Edited:   /usr/local/blackboard/content/vi/BBLEARN/plugins/bb-stream/webapp/WEB-INF/jsp/stream/streamSettings.jsp
and changed line: <div class='scrollable' bb:lowestElement="streamSettingButtons" bb:dynHeightScroll="true">
to: <div class="scrollable scrolling" bb:lowestelement="streamSettingButtons" bb:dynheightscroll="true" style="height: 128px; overflow-y: hidden;">

Cycled all nodes with:  "PushConfigUpdates.sh"

05/30/2013, 5:00AM - 5:30AM
Added the following (all nodes) to bb-config.properties / bbconfig.jvm.options.extra.tomcat:

  • -XX:+AggressiveOpts
  • -XX:+UseCompressedStrings
  • -XX:+UseCompressedOops
  • -XX:+OptimizeStringConcat
  • -XX:+UseStringCache

And to bb-config.properties / bbconfig.jvm.options.gc

  • -XX:+UseParNewGC

Cycled all nodes with:  "PushConfigUpdates.sh"

05/24/2013, 2:00AM - 2:10AM
Stopped all services.  Removed /usr/loca/blackboard/content/vi/BBLEARN/activemq folder to trigger the recreation of ActiveMQ's tablespace.  Truncated table "peer_service" and then restarted BbLearn services on all nodes, one at a time.

05/23/2013, 2:00AM - 6:00AM
Upgraded to SP12, CP1.  Installed B2s:  software updates, cloud and profiles, via B2Manager.sh.  Updated all other B2s which had a new version available.

04/26/2013, 2:00AM - 3:58AM
Storage is now using the new iSCSI switch.  XenServer patch XS602E013 was applied to fix NFS server issue.

04/25/2013, 2:00AM - 6:00AM
Weekly maintenance window:  The first part of a conversion to a new iSCSI switch was implemented.  All disks on the Storage Link storage repository were migrated to a standard LVM over iSCSI SR.  (This type of SR is known to work with this version of XenCenter.)  During an outage on Friday (4/27), the final cut over and re-ip of the storage array took place.

04/04/2013, 2:00AM - 6:00AM
Weekly maintenance window:  January 2013 Oracle PSU installed on LEARNPRD.  All VMs rebooted, all nodes cycled.

03/12/2013, 8:33PM - 9:02PM
Lost network connectivity to bryce and all learn nodes.

03/09/2013, 8:50PM - 9:20PM
The NFS server lost its network connection.  The network adapter had to be reset in order to restore connectivity.

03/07/2013, 5:00AM
Weekly maintenance window:  LRNSI-10005 and LRNSI-10147 applied.  All nodes cycled (./PushConfigUpdates.sh).

03/05/2013, 5:00PM
admin.bblearn was taken out of the pool and is no longer public facing.

02/28/2013, 5:00AM - 6:00AM
During weekly maintenance, all the nodes were cycled (./PushConfigUpdates.sh).

02/09/2013, 7:15PM - 7:54PM
The disk attached to the NFS server entered a read only state after some network issues.  This is a known issue with XenCenter and is under investigation for a long term fix.

01/30/2013, 3:00PM
TIPS (Technology: Information Proficiency Skills) turned off in BbLearn.

01/17/2013, 5:00AM - 6:00AM
During weekly maintenance, all the nodes were cycled (./PushConfigUpdates.sh).

01/10/2013, 2:20AM - 6:00AM
During weekly maintenance, all the VMs were rebooted and all BbLearn nodes were restarted.  nfs.bblearn.nau.edu was also rebooted.

01/08/2013, 7:15PM
The new theme:  "Bb Learn 2012" was applied.

01/06/2013, 12:20PM - 1:27PM
The NFS server lost connection to the disk array.  Rebooted NFS server and services are now restored.

01/02/2013, 11:30AM
The Student Support tab was added.

12/20/2012, 02:00AM - 06:00AM
During weekly maintenance window, patch LRNSI-9661 was applied:  (student's My Grades screen shows the Alignments column and the button in every row even when there are no goals set in Tools).

11/29/2012, 02:00AM - 06:00AM
During weekly maintenance window, the October PSU from Oracle was applied to LEARNPRD.

11/01/2012, 02:00AM - 06:00AM
During weekly maintenance window, patch LRNSI-9233 was applied:  (When students in a group try to access the collaboration tool in their group they get the access denied error).

10/25/2012, 02:00AM - 06:00AM
During weekly maintenance window, patch set 2 (SP9) and patch set 3 (SP9) were applied.  LRNSI-9455 was also reapplied after patch set 3, as it had been updated since its' inclusion in patch set 3.  This script was also run in LEARNPRD: rollup_backfill.sql.  What's included in patch set 2 and patch set 3 are here.

10/05/2012, 06:48AM - 08:48AM
BbLearn was unavailable due to contention on the NFS server after an upgrade to the firmware on the disks.  Several jobs on the NFS servers were stopped which restored service.

09/14/2012, 12:00PM - 12:00PM
BbLearn was unavailable due to the virtual NIC on the NFS server having lost connectivity.  No content could be retrieved which brought the system down.

09/06/2012, 9:00PM - 09/07/2012, 8:50AM
BbLearn was available for most read only tasks, but when users tried to upload files, save grades or perform other functions in BbLearn which required a write to disk, they recieved a message:  "No Space Left on Device."  The cause was Apache logs had not rotated off of 4 of the nodes:  prd1, prd3, prd4 and prd5; prd2 was fine.  Backup data was archived by hand ./logs/archives, freeing up disk space.

08/23/2012, 7:00PM - 7:50PM
BbLearn was unavailable due to the virtual NIC on the NFS server having lost connectivity.  No content could be retrieved which brought the system down.

08/23/2012, 2:00AM - 6:00AM; regularly scheduled maintenance
New batteries installed in Bryce.  Oracle July 2012 PSU applied to LEARNPRD

08/18/2012, 5:30AM
Applied patch LRNSI-8214 to address the bug:  "Thousands of "Timer-xxxxx" Being Leaked Causing Descriptor Exhaustion and Server Lock-up.

08/17/2012, 4:00PM
Added the following indexes to help with queued tasks and course copies:

  • create index EUD_ITEM_RECIPIENT_BMH_1 on EUD_ITEM_RECIPIENT(EUD_ITEM_PK1,TYPE,USER_PK1) online tablespace BB_BB60_INDX;
  • create index COURSE_USERS_BMH_1 on COURSE_USERS(CRSMAIN_PK1,ROW_STATUS,ROLE) online tablespace BB_BB60_INDX;
  • create index EUD_ITEM_ROLE_BMH_1 on EUD_ITEM_ROLE(EUD_ITEM_PK1,ENTITLEMENT_ENTITLEMENT_UID,TYPE) online tablespace BB_BB60_INDX;
  • create index EUD_ITEM_BMH_2 on EUD_ITEM (DATA_PENDING_IND,IMPORTANT_IND,CRSMAIN_PK1) online tablespace BB_BB60_INDX;
  • create index EUD_METHOD_SETTING_BMH_3 on EUD_METHOD_SETTING(SOURCE_TYPE,EVENT_TYPE, METHOD_TYPE) online tablespace BB_BB60_INDX;

08/17/2012, 2:55PM
Set "Notification Cleanup" (System Admin -> Tools and Utilities -> Notifications) to 60 days in order to minimize notification traffic.  (was set to 120 days).

08/16/2012, 6:00AM
Upgraded to SP9, patchset 1.  Patchset 1 includes (LRN-55438, LRN-52073, LRN-51203, LRN-51725, LRN-50582, LRN-49207, LRN-46952, LRN-50259, LRN-49812, LRN-52127, LRN-48738, LRN-53498, LRN-53358, LRN-49868, LRN-51676, LRN-44772, LRN-49819, LRN-48404, LRN-54799, LRN-55374, LRN-54814)

07/06/2012, 7:40AM
admin.bblearn quit around 3:30AM (7/6/2012) and was force rebooted at 7:40AM.  Service is restored.

06/14/2012, 2:00AM
Applied April 2012 Patch Set Update (PSU) to DB LEARNPRD

05/24/2012, 1:30PM
WileyPlus building block upgraded to the latest version (2.0.91700381)

05/17/2012, 11:00AM 
Building Blocks upgraded to their latest versions:
   Bb Mobile Web Services  v91.3.1
   SafeAssign  v2.5.0
   NBC Content  v3.0.22

05/11/2012, 3:30PM
Flashback was turned back on in LEARNPRD
backup table BBLEARN.ACTIVITY_ACCUMULATOR_OLD was deleted

05/07/2012, 9:25PM - 10:05PM, BbLearn service interruption
BbLearn was down for around 40 minutes due the the EqualLogic disk array running out of available space

05/07/2012, 8:30AM
NFS recommendations from BB Support were applied (no down time was required)
   at the clients (prdx.bblearn.nau.edu), file system table settings were edited/added (/etc/fstab):  bg,noacl,noatime,nodiratime,rsize=32768,wsize=32768
   at the server (nfs.bblearn.nau.edu), in /etc/sysconfig/nfs, chaned RPCNFSDCOUNT to 32 (was set at 8)

05/05/2012, 2:00AM
BB patch LRNSI-5668 was installed.  Service was restored at 2:10AM

LRNSI-5668:  Header Injection Vulnerability in Content Collection Could Allow Tampering & High application wait and user IO since upgrade to SP6 - Causing Frequent Outages & address the full scans on XYF_FILES and XYF_URLS by hinting the getAvailableQuota() query for Oracle; also fix FileSystemUtil & Xythos DB load spikes around SUMVIEW DIGESTVIEW queries & change Quota query to be  more performant(XS-18015, XS-18016) &  Download content from WFS home with many files by XD fails and CPU usage is 100% (XS-18013). Jars are from Xythos Build: h72SP2_7_2_91_1119

05/04/2012, 7AM - 3:30PM
service was slow all day due to jam up on the NFS server.  Machines lmsvm1, lmsvm2 and lmsvm3 were rebooted at 2:00PM and service was restored at 3:30.  Changes were made systems configuration file "fstab", which now is set at: 

nfs.bblearn.nau.edu:/lmsStorePrd/PRD-LEARN-Content/learnContent /usr/local/blackboard/content   nfs     soft,intr,rw,rsize=65536,wsize=65536,noatime,actimeo=1 0 0
Changes made include:  doubling the size of rsize and wsize, adding noatime and actimeo.

05/03/2012, 11AM - 12AM
3 more groups of redo logs (2GB each) were added to LEARNPRD

05/03/2012, 2AM - 6AM, weekly maintenance window
purge_accumulator was disabled before its nightly run (1AM)
all nodes shutdown
table "BBLEARN.ACTIVITY_ACCUMULATOR" was rebuilt using export - import; pruned down now to around 70 million rows
all nodes brought back up; purge_accumulator job started
DB standby was also turned off for a testing experiment; BB ran a stress test around 3:50AM.  Standby was turned back on at 7:26AM and was caught up by 8:16AM

05/02/2012, 4:30PM, tuning before final exam week
commented out SQL statement:  "delete from BBLEARN_stats.activity_accumulator ..." in Oracle procedure "BBLEARN_STATS.SYNC_ACTIVITY_ACCUMULATOR"
commented out DiskUsage task from /config/bb-tasks.xml and /config/bb-tasks.xml.bb
restarted admin.bblearn node for task change to take effect

05/02/2012, 8:56AM, prd5 restart
prd5 did not come up cleanly in the cluster restart earlier this morn (errors in bb-services-log.txt:  "Could not initialize class ...") and had to be restarted

05/02/2012, 2:15AM, cluster restart
full cluster was restart to see how this would impact stress testing through the load balancer

04/27/2012, 1:00AM, purge_accumulator now running with a new "days to keep" argument
"days to keep" changed from 180 to 90 for the purge_accumulator job

04/26/2012, 9:40AM, prd4.bbearn.nau.edu Network connectivity restored
prd4.bblearn.nau.edu VM had been in some funky network state (we've seen this before with other nodes), and the NFS service had to be restarted.

04/24/2012, 02:00AM, Flashback turned off and patch applied
flashback was turned off in LEARNPRD (since 4PM, 04/23)

BB patch LRNSI-7129 was applied to help alleviate problems with long running query against tables "xyf_files" and "xyf_urls"
   LRNSI-7129:  Prevent 360 degree view report from listing children for top level directories
     Resolved bugs:  LRN-51675

04/18/2012, 02:00PM, BB Support suggested maintenance
backed up table CMS_FILE_ACCESS and truncated it (27,976,468 rows)

Also, noted that 2 images:  Android-m.nau.edu.png and iOS-Blackberry.png were still being loaded out the of content system.  They are now being served from the nodes.

04/10/2012, 02:00AM, cluster restart
BBLearn cluster was restarted (PushConfigUpdates) for changes made in the afternoon of 04/09/12 (maxclients 500 -> 200) to take effect.

04/10/2012, 12:06AM, prd1.bbearn.nau.edu VM rebooted
prd1.bblearn.nau.edu VM lost network connectivity which required a reset of the NIC via Xen and a restart.  Service was restored at 12:12AM

04/09/2012, 4:30PM, changes made to PRD environment as recommended by BB support/audit (Stephen Byrne)

JDBC settings for CMS document store (via GUI) were changed from:
   Maximum Connections:  25 -> 100
   Minimum Connections:    2 -> 25

added to /etc/sysctl.conf: 
   net.ipv4.tcp_keepalive_time = 1800
   fs.file-max = 65536

added to /etc/secutiry/limits.conf:
  *               soft     nofile          65536
  *               hard     nofile          65536

added to /etc/bashrc:
  ulimit -Hn 65536
  ulimit -n  65536

changed bbconfig.appserver.max.spare.threads, bbconfig.appserver.maxthreads and bbconfig.unix.httpd.maxclients in bb-config.properties from 500 to 200

changed MaxClients in /usr/local/blackboard/apps/httpd/conf/httpd.conf from 500 to 200

04/05/2012, 8:45AM, prd2.bbearn.nau.edu VM rebooted
prd2.bblearn.nau.edu VM had been in some funky state since 12:48AM and had to be force rebooted.  prd2 was back up and BBLearn available on prd2.bblearn.nau.edu at 8:47AM.  End users were not affected.

04/04/2012, 3:14PM, admin.bblearn started and then PushConfigUpdates (3:27PM)
admin.bblearn was not running, and was started.  Perhaps it never came back up from PushConfigUpates this morn?  But, a simple start would not get the exec service running.  Finally did a PushConfigUpdates.sh and admin is now responsive.

04/04/2012, 9:05AM, admin.bbearn.nau.edu bounced
Huge DB spike (8:49AM - 9:11AM) exhausted all available sessions (2500) in the LEARNPRD DB.  admin.bblearn.nau.edu was bounced (PushConfigUpdates) in order to stop ongoing processing in hopes of reclaiming DB sessions.  One more spike occurred (9:25AM - 9:38AM). All quiet since.

There were also 2 "normal" (Abbie-Normal? (wink)) spikes earlier in the morn:  4:48 - 4:56AM and 5:24 - 5:31AM.

04/03/2012, 6:10PM, admin.bbearn.nau.edu bounced
admin.bblearn.nau.edu was bounced (PushConfigUpdates) so purge accumulator would only run once (change had been made to bb-tasks.unix.xml; see previous log entry).  Morning of 4/4 showed only 1 instance of Purge Accumulator running, so all's good.

04/03/2012, 3:10PM, Changes to bb-tasks.unix.xml and bb-tasks.unix.xml.bb
Last Thursday (03/29/2012), when nodes were bounced, admin node "bb-tasks.unix.xml" reverted back to running both Purge Accumulator and Rotate logs because the update (commenting out of jobs) had not been made to "bb-tasks.unix.bb.xml."  Both files have now been changed (both jobs commented out in xml files) and admin.bblearn will be bounced soon so changes will take effect before tonight.

04/03/2012, 10:30AM, Full Text Indexing time limit
Set a 60 minute time limit in "fullTextIndexDaily.sh" in "/home/bbuser/fullTextIndexing" to prevent multiple runs from occurring.  Previously, this job had no time limit.

04/03/2012, 9:38AM, Probe Change for Load Balancer
The health check probe for the load balancer was changed from /index.html to /webapps/portal/healthCheck and expected status changed to 200 299

04/03/2012, 9:38AM, admin.bbearn.nau.edu bounced
admin.bblearn.nau.edu was bounced to clean it out and relieve burden on DB.  Memory on admin.bblearn.nau.edu was increased to 18GB.  BB consultant was having trouble running snapshot scripts to create fake courses for load testing.  admin.bblearn.nau.edu was back up at 9:44AM.

03/30/2012, 2:00PM, Login page images moved
The custom images which load on the login page were moved out of the content area and into /usr/local/blackboard/content/vi/BBLEARN/branding/login (on all nodes) where they can be served by apache.  This was recommended by BB Support as part of the DB Spike ticket.

03/29/2012, 4:29PM, lookup_link_target_icon_url patch applied
A recursive link was created in a class that caused DB performance issues. This is a known issue (LRNSI-2710).  Bb support provided an updated procedure for "lookup_link_target_icon_url." All sessions stuck using the old version of the procedure were killed so the new procedure could be put into place. The system was slow, but never became unresponsive.  Other than users being unable to access the class in question (1121-5894), end users should not have been effected.

03/29/2012, 1:26PM, Admin node bounced
The admin node was bounced in hopes of clearing a heavy DB load.

03/29/2012, 5:45AM, Weekly maintenance window restart
All nodes restarted (PushConfigUpdates.sh).  The admin node was back up at 5:20AM.

03/28/2012, 10:25AM, WileyPlus Building Block upgrade
The WileyPlus Building Block was upgraded to v1.1

03/26/2012, 3:15PM, restart admin
Restarted admin because of a weird network disconnect from Xen.

03/26/2012, 1:00PM, prd5.bblearn restart
Prd5 had crashed on 3/25 at 11:30AM and was rebooted to be brought back into service.

03/26/2012,  11:30AM, setting changes

  • in bb-tasks.unix.xml, commented out both "bb.stats.purging" (purge accumulator) and bb.log.rotation" since both are controlled on admin.bblearn by crontab
  • in bb-config.properties, set both "bbconfig.appserver.max.spare.threads" and "bbconfig.appserver.maxthreads" to 500, to conform to "bbconfig.unix.httpd.maxclients" (Apache)
  • in bb-tasks.xml, changed the "delay" parameter in the "DomainCollectionTask" from "<property name="delay" value="1000" />" to "<property name="delay" value="21600000" />"

03/26/2012,  9:30AM, setting change
Changed System Admin -> Feature and Tool Management > Feature and Tool Availability > Statistics Available to "No" as advised in BB Case ID:  935025 regarding table "CMS_FILE_ACCESS"

03/24/2012,  1:17PM, Admin Restart.
Learn was running slow due to many UPDATE sessions SET timestamp = :1 WHERE session_id = :2 running. Restarted Admin server to see if it would help this issue. No end users should have been kicked off.

03/23/2012  4:56PM, Admin node restart
The admin node was restarted as it was doing full gc.  End users should not have been affected.

03/15/2012  4:00 AM:  Weekly maintenance window
LEARNPRD was upgraded to Oracle 11.2.0.3.  All nodes were started back up with PushConfigUpdates.shThe admin node was back up at 3:37AM.

03/14/2012  7:45 AM:  admin node bounced
The BbLearn admin node was bounced (shutdown followed by "PushConfigUpdates.sh") to see how it would effect the LEARNPRD spikes.  End users were not effected.

03/09/2012  12:15 PM:  Truncating Sessions and sessions_inds and rebuilding indexes
DB spikes since 5:40AM were hindering system performance.

All BbLearn nodes were shutdown.
The following DDL was executed:

 Push config was executed to bring the servers back up.

03/08/2012  5:30 AM:  Weekly maintenance window
All BbLearn nodes were cycled (./PushConfigUpdates.sh) thus invoking new setting in config/tomcat/classes/Search.properties of EventBasedIndexingEnabled= false.

03/07/2012  09:55 AM:  change to time to rotate logs and indexing param change
Changed time to rotate logs to 1:00AM because the McGraw Hill B2 was rotating its log at the same old midnight time and was causing the delivered Bb script to fail.
Update config/tomcat/classes/Search.properties[.bb] and set EventBasedIndexingEnabled=false. This will not take place until the restart planed for the morning. 

03/07/2012  09:55 AM:  change to run time of create-all-courseorg-directories.sh
create-all-courseorg-directories.sh is run daily to create a folder for every course so the library folks have a place to put library reserve content.  This job runs about 2 hours and was (pre 3/7) scheduled to run at 11PM (via crontab, user bbuser, admin node).  It is now set to run at 2AM.

03/06/2012  10:15 AM:  full cluster restart
BbLearn was unresponsive.  Did a full cluster restart.  This error was in bb-services-log.txt: "Failed to run cache update. - java.lang.NullPointerException" on all nodes, after the full cluster restart.  Ran PushConfigUpdates.sh on prd1.bblearn which seems to have cleared up the problem.

03/02/2012  4:00 PM:  admin node bounced
The BbLearn admin node was bounced to see how it would effect the LEARNPRD spikes we've been seeing since we bounced the cluster yesterday morn.  End users were not effected.

03/01/2012  5:30 AM:  Weekly maintenance window
All BbLearn nodes were cycled (./PushConfigUpdates.sh) and patch LRNSI-6358 was applied.

02/15/2012  10:00AM - 11:05AM:  BbLearn slowdown/unavailable
BbLearn was slow and unresponsive during the morning, which could have been partially due to a full backup running.  All nodes and LEARNPRD were cycled.  SGA memory was increased from 8GB to 12G and the PGA memory was increased from 4GB to 6GB.  Ticket was opened with BbSupport and AWR report was enclosed.

02/09/2012 06:00AM Normal LMS maintenance window.
LEARNPRD temp tablespace was resized to 16GB.  The number of database processes was increased from 1600 to 2500 in light of 2 new nodes (prd4.bblearn and prd5.bblearn) being added.

02/05/2012 11:00AM - 1:42PM; BbLearn was down/unresponsive due to LEARNPRD being under heavy load
All nodes and LEARNPRD were restarted due to heavy DB load.  TEMP table space was filled.  Bb Patch LRNSI-5672 was applied (latest Xythos patch).  This seems to be a new problem, as the DB blocking lockers are different from our last DB incident.

02/03/2012 02:45PM PRD3 Restarted
PRD3 was restarted as it was doing full garbage collection.

02/03/2012 11:30AM PRD3 Restarted
All nodes were swamped, as load was heavy.  LEARNPRD was OK.  PRD3 was restarted to bring it back from beyond.

2 new nodes were added to the cluster, prd4.bblearn and prd5.bblearn.

01/26/2012 06:00AM Normal LMS maintenance window.
Bb patch LRNSI-5808.bbp applied.

Added to /apps/httpd/conf/ssl.conf.bb and /apps/httpd/conf/ssl.conf:
   SSLCertificateChainFile /usr/local/blackboard/content/ssl/nau.chain

Updated /config/bb-config.properties:
  bbconfig.unix.httpd.maxclients=500

01/25/2012 01:07AM - 2:30PM BbLearn performance sporadic; Production nodes and LEARNPRD bounced.
LEARNPRD bounced due to huge number of blocking lockers.  During down time, tempspace was resized from 32GB to 30GB

01/24/2012 10:56PM Admin node bounced.
Admin node doing full garbage collection and was restarted.  This should not have effected users.

01/23/2012 07:20AM - 7:50AM Production nodes and LEARNPRD bounced.
LEARNPRD bounced due to huge CPU spike on bryce as well as sustained I/O waits and blocking lockers

01/19/2012 06:00AM Normal LMS maintenance window.
mod_gzip turned off on all Apache web servers across all nodes and Apache restarted.

01/17/2012 08:50PM BbLearn not responding.
Apache died across all nodes.  Apache was restarted and admin.bblearn was added to the pool which now includes 4 nodes  Service was restored by 9:20PM.

01/12/2012 07:00AM WileyPLUS/Blackboard Integration Building Block Installed on Bb Learn.
The building block to integrate WileyPLUS with Bb Learn was installed and configured.

01/12/2012 06:00AM Upgrade to Bb Learn 9.1 Sp7 HF1 Patch Set 1
Learn was upgraded to the latest version. This included a Service Pack, a Hot Fix and a Patch Set. As of now the following patches are installed:

LRNSI-5950  2012-01-12 03:13:14 -0700   2011-12-14 14:09:20 -0700   LRN-39265  
LRNSI-5955  2012-01-12 03:12:33 -0700   2011-12-14 14:23:14 -0700   LRN-28887  
LRNSI-5952  2012-01-12 03:13:28 -0700   2011-12-14 14:13:24 -0700   LRN-41624  
LRNSI-5946  2012-01-12 03:13:48 -0700   2011-12-14 13:58:45 -0700   LRN-43622  
LRNSI-5959  2012-01-12 03:11:53 -0700   2011-12-14 16:18:02 -0700   LRN-31984  
LRNSI-5957  2012-01-12 03:12:08 -0700   2011-12-14 16:31:51 -0700   LRN-30399 
LRNSI-5951  2012-01-12 03:12:46 -0700   2011-12-16 11:34:43 -0700   LRN-28742, LRN-33359
LRNSI-5961  2012-01-12 03:11:36 -0700   2011-12-14 16:21:33 -0700   LRN-44451  

During the maintenance window, the following changes were made on bryce:

1. Increased the size of /dblog1 and /dblog2 from 10GB to 50GB

2. Made Kernel changes so dirty pages in memory are written out more frequently:

  • vm.dirty_background_ratio = 3
  • vm.dirty_ratio = 15
  • vm.dirty_expire_centisecs = 500
  • vm.dirty_writeback_centisecs = 100

3. Changed the kernel parameter that determines how aggressive swapping is done.  Set to 0 so swapping is only done when needed.

  • vm.swappiness = 0

4. Configured hugepages for Oracle which makes the memory pagesize larger for Oracle processes (this memory is not swappable)

  • vm.nr_hugepages = 6656
  • vm.hugetlb_shm_group = 101
  • ulimit settings for Oracle:
  • oracle soft memlock 13631488
  • oracle hard memlock 13631488

The Following changes were made to LEARNPRD:

  1. set SGA_TARGET to 8G
  2. set PGA_AGGREGATE_TARGET to 4G
  3. reset MEMORY_TARGET
  4. set USE_LARGE_PAGES to “only” (ensures that the allotted space is available on database startup)


01/10/2012 08:20AM chksum turned off in LEARNPRD
The db_block_checking parameter was set to "FALSE" (was previously set to "TRUE").

01/10/2012 07:25AM PRD3 Restarted
PRD3 was restarted in an attempt to clear grade book long sessions (UPDATE GRADEBOOK_MAIN SET ...).

12/22/2011 05:00AM Scheduled Maintenance
Apply Oct2011 Oracle PSU to LEARNPRD.bryce and LEARNPRD.chaco

12/15/2011 10:05PM Added another node
Added PRD3 back into the cluster.

12/14/2011 12:30PM Full Restart
BbLearn, LearnPRD and bryce were bounced alleviate heavy load on LEARNPRD (bryce).  BbLearn cluster was brought up on only 2 nodes, PRD1 and PRD2.  Service was restored by 1:25PM.

12/14/2011 11:10AM Full Restart
BbLearn and LearnPRD were bounced alleviate heavy load on LEARNPRD (bryce).  Service was restored by 11:38AM.

12/14/2011 7:30AM Full Restart
BbLearn was bounced alleviate heavy load on LEARNPRD (bryce).  Service was restored by 8:05AM.

12/13/2011 1:45PM Full Restart
BbLearn was bounced alleviate heavy load on LEARNPRD (bryce).  admin node was added as a fourth node to the cluster.  Service was restored by 2:15PM.

12/12/2011 6:30PM Full Restart
BbLearn and LEARNPRD were shut down and restarted to alleviate an indeterminate load on LEARNPRD (bryce).  Service was restored by 7:55PM. 

12/11/2011 8:30PM Full Restart
After a rolling restart, BbLearn was shut down and restarted to alleviate an indeterminate load on LEARNPRD (bryce).  Service was restored by 8:45PM.  Later (around 11PM), service again became sluggish.  LEARNPRD had all of its 32GB of temp space full.  BbLearn became stable again around 11:30 PM (perhaps after enough users gave up?).

12/04/2011 7:30PM Rolling restart and Full Restart
After a rolling restart, BbLearn was shut down and restarted to alleviate an indeterminate load on LEARNPRD (bryce).  Service was restored by 7:45PM.

11/21/2011 10:30PM Rolling restart
Due to heavy load, service was unavailable or inconsistent.  Error messages varied; some pages loaded, others loaded partially, some not at all.  A rolling restart was performed on all nodes. Service was restored by 11:25PM.

11/07/2011 10:30PM Full Cluster restart
Problems with prd3 forced the other two nodes to take too many connections forcing a full restart. Service was restored by 10:50PM.

11/03/2011 02:50PM PRD1 disk at 100%
prd1.bblearn's drive mount partition which holds content (logs) reached 100%.  Users connecting to prd2 and prd3 were not affected.  Users on prd1 attempting to upload content did recieve an error message.  Log files were off-loaded to alleviate the problem.  Full service was restored by 3:25 PM.

10/31/2011 07:50AM PRD2 restarted
prd2.bblearn had been unreachable since 7:46 PM, Sunday 10/29, and was restarted.

10/20/2011 02:00AM Scheduled Maintenance
All Bb Learn application servers were restarted.   

10/06/2011 02:00AM Scheduled Maintenance
rollback LRNSI-4851(Original Quiz Access Denied Patch) , apply 886515C8R1LRNSI-4851_v4.bbp(Quiz Access Denied Patch), apply LRNSI-5098.bbp(Fix PDF in OSX)

09/22/2011 02:00AM Scheduled Maintenance
All nodes including NFS, admin and prd1-3 were restarted. New Xythos patch applied via ./bbpatch.sh apply 347146_LRNSI-4148.bbp.

09/19/2011 10:40AM NFS Parameter change
The noatime parameter was set for the learnContent mount point to improve NFS performance.

09/15/2011 2:00AM-6:00AM Scheduled Maintenance

  • BbPatch was upgraded to 2.0 on all app nodes
  • Patch AS-162528 was applied to fix pdf rendering issues within learning modules
  • Patch LRNSI-4851 was applied to fix accedd denied errors that were showing up while taking quizzes

09/13/2011 10:00AM Performance Tuning

  • cursor_sharing was set back to exact

09/01/2011 3:00PM Performance Tuning
Based on recommendations from Blackboard (ticket #882730) some parameters were adjusted to improve performance.

  • analyze_my.bbtabs and dbms_utility.analyze_schema('SYS','COMPUTE'); db jobs were set to broken
  • The NotificationRemoveStaleDataTask was set to run once per day
  • The DistributionSendNotificationTask and DistributionSendNotificationTask were set to run once per hour
  • cursor_sharing was set to similar

09/01/2011 2:00AM-6:00AM Scheduled Maintenance
JVM tuning and patches applied. Each node was bumped up to 8 VCPU's. The heap and permgen were increased from 10GB, 512M to 12GB,1024M respectfully. and  The ¿XX:ParallelCMSThreads and XX:ParallelGCThreads parameters were bumped from 2, 4 to 3, 8 respectfully. A one-off Xythos patch was applied along with patch AS-162462.bbp that resolves AS-162460, Connection leak in CourseEnrollmentManagerImpl$CheckDomainUserQuery due to incorrect use of JdbcQueryHelper.

08/25/2011 2:00AM-6:00AM Scheduled Maintenance
The Learn database server had memory added to it and the MEMORY_MAX_TARGET was bumped to 12GB.

08/04/2011 2:00AM-6:00AM Scheduled Maintenance
All nodes were bumped up from 2 vcpus to 4 vcpus. An updated my courses module was installed. Timeout values were fixed.

07/28/2011 2:00AM-6:00AM Scheduled Maintenance
Bb Learn was upgraded to service pack 6

06/30/2011 2:00AM Scheduled Maintenance -- Disk Resize, Timeouts and Memory
All nodes have had the blackboard install moved to its own partition. All timeouts where changed to 120 mins. Admin had its memory increased. 

06/24/2011 6:05AM prd1 restart
prd1 was restarted because it was using 100% of its CPU for several days. This should not have affected end users.

06/23/2011 2:00AM-6:00AM Scheduled Maintenance
The main tomcat session timeout was changed fromt the default of 30 minutes to 120 minutes on prd2. Also, the stack memory was changed from 320Kb to 1024Kb on prd2. This was done on prd2 only to guage the impact of memory usage and performance of making this change across the cluster. prd1 and prd2 were then restarted.

06/20/2011 7:20AM prd2 restart
prd2 was restarted because it was using 100% of its CPU for several days. This should not have affected end users.

06/16/2011 2:00AM-6:00AM Scheduled Maintenance
Applied Blackboard Learn patch BUG:AS-158601 PATCH:AS-160257 - Cross-site Scripting Vulnerability in Blackboard Learn Discussion Board Could Allow Elevation of Privilege and Xen Server hotfix XS56E006 to the Xen cluster.

06/09/2011 2:00AM-6:00AM Scheduled Maintenance
Full text indexing was disabled from the GUI and immediate indexing was disabled for Xythos as per Bb Support's instructions. Full text indexing was then scheduled to run via command line script through cron.

06/06/2011 12:20AM prd2 restart
prd2 was restarted because it was using 100% of its CPU for several days. This should not have affected end users.

06/06/2011 12:15AM prd1 restart
prd1 was restarted because it was using 100% of its CPU for several days. This should not have affected end users.

06/02/2011 5:00PM-6:15PM SSL Outage
The SSL certificate that was hosted on the nodes expired. This caused the ACE to stop trusting the nodes. The cert was updated on the ACE, but since the ACE plays man-in-the-middle it still needed to have a valid, non-expired cert on the nodes. The cert was updated and the nodes restarted.

06/02/2011 2:00AM Upgrade to SP5
BBLearn was upgraded to SP5 on all nodes. Patches BUG:AS-154719 PATCH:AS-159089.bbp, BUG:AS-155049 PATCH:AS-159092.bbp and BUG:AS-155143 PATCH:AS-159210 applied. Memory on prd1.bblearn.nau.edu and prd2.bblearn.nau.edu upgraded to 14Gig from 12Gig. Theme also updated to remove blue arrows.

05/26/2011 2:00AM New SSL Cert and Oracle CPU
Oracle was updated to the latest CPU. A new SSL cert was installed. This new cert is using the NAU global sign CA. 

04/25/2011 12:00PM LMSfeed 1.7.1 deployed
A new version of the integration feed was deployed that should speed up operations.

04/11/2011 2:00AM - 6:00AM Scheduled Maintenance
The memory_max_target on the production database for LEARPRD was decreased by 1GB to alleviate the excessive swapping that was happening on Bryce. This should improve performance.

04/04/2011 12:00PM LMSfeed 1.7 deployed
A new version of the integration feed was deployed. This new feed allows for combined sections in Bb Learn.

03/24/2011 - Scheduled Maintenance - Cluster Reboot
The cluster was restarted to allow the new kernel updates to take effect.

03/18/2011 9:00AM - Restart and memory increase
Both production VM's had their available memory increased to 12GB each and then restarted in turn. This should not have affected end users.

03/14/2011 10:25AM - BBLEARN_STATS cchema altered
The following sql was executed to fix a schema mismatch that was causing the PurgeAccumulator job to fail.

03/03/2011 - Scheduled Maintenance - Cluster Reboot
The cluster was restarted to allow the new kernel updates to take effect. 

02/28/2011 - RedHat Patches Applied
No downtime. All updates from RHN have been applied to the Learn cluster. New kernel will be used next reboot.

02/17/2011 1:30PM - Disables eReserves tool
The eReserves tool is dysfunctional to the point that we are not going to use it, so the tool was disabled from the content collection.

02/17/2011 8:35AM - New NAU theme made default
A new NAU theme was uploaded and made the system default. The only change to this theme from the Bb Learn theme at this point is that the course title listed in the top of the course menu is now hidden.

02/17/2011 8:30AM - Modified custom login page
Small change was made to custom login page to change the link to e-Learning from www.nau.edu/elearning to www.nau.edu/elc so that it would fit in the page column width. New page was uploaded to prd1 and prd2.

02/17/2011 2:00AM-6:00AM - Scheduled Maintenance
Oracle upgraded to 11.2.0.2 and January CPU applied.

02/15/2011 8:58AM - pinned my courses module
The MyCourses module was removed from the default tab layout and then added back by pinning it to the top center column. All users should now see the MyCourses module at the top of the center column on the Home tab and now cannot minimize or move that module.

02/13/2011 2:04PM-2:17PM - bblearn down due to database crash
The database that is used by bblearn.nau.edu crashed which caused bblearn to be inaccessible for a time. The database was restarted and Bb Learn came back online.

02/10/2011 10:05PM-10:50PM - bblearn down due to database crash
The database that is used by bblearn.nau.edu crashed which caused bblearn to be inaccessible for a time. The database was restarted and checked for proper health before restarting the bblearn service.

02/10/2011 2:00AM-6:00AM - Scheduled Maintenance

  • Blackboard patch AS-155324 was applied to fix a quiz submission issue
  • The system-wide session timeout was changed from 3 hours to 2 hours
  • The assessments tool timeout was changed from 20 minutes to 2 hours
  • System tasks were disabled on prd2.bblearn.nau.edu to help improve overall performance and stability

01/21/2011 3:40PM - prd1 restart and permgen settings change
prd1.bblearn.nau.edu ran out of PermGen space and needed to be restarted. This should not have affected end users. The permgen setting was changed from 256MB to 512MB and the pushupdates script was run so that it will take effect.

01/27/2011 2:00AM-6:00AM - Scheduled Maintenance
The oracle parameter, FILESYSTEMIO_OPTIONS=SETALL was remove from Bryce for LEARDPRD and VISTA4. LEARNPRD and Vista4 were then restarted.

01/21/2011 2:30PM - prd1 restart
prd1.bblearn.nau.edu ran out of PermGen space and needed to be restarted. This should not have affected end users.

01/18/2011 11:50AM - prd2 restart and permgen settings change
prd2.bblearn.nau.edu ran out of PermGen space and needed to be restarted. This should not have affected end users. The permgen setting was also changed from 256MB to 512MB to try to keep this from happening again.
prd1.bblearn.nau.edu was also changed from 256MB to 512MB which will take effect the next time it is restarted.

01/12/2011 7:00AM - 8:00 AM - Upgrade
Learn was upgraded to (9.1 sp4). Prd2 was also configured but not yet added to the server farm.

01/03/2011 8:00AM - 5:00 PM - Scheduled Maintenance
The new installation (9.1 sp3) and database were brought online as bblearn.nau.edu.

  • No labels