Skip to end of metadata
Go to start of metadata

Vista Change Log

Please include each and every change that is made to any of the application servers, database servers, load balancer, or feed/feed database.

DBA Vista Changelog

March 24, 2012:  1:17PM, Admin Restart.
Learn was running slow due to many UPDATE sessions SET timestamp = :1 WHERE session_id = :2 running. Restarted Admin server to see if it would help this issue. No end users should have been kicked off.

March 23, 2012:  9:00 - :9:20 AM, Vista was down.
Oracle January PSU applied to "vista4.chaco.vista.nau.edu"

February 15, 2012
Vista bounced
Vista was down from 2:00PM through 2:15PM for a restart.  File "config.xml" was messed up having migratory targets for JMS pointing to servers which no longer exist.

December 12, 2011
Bryce bounced
Vista was down from 12:50PM through 1:40PM for a restart of bryce, as BbLearn had swamped the machine.

June 30, 2011
Google analytics
Google analytics code was added to the login page, primarily to track referrers to begin to identify and eliminate links into vista.nau.edu for the fall.

06/30/2011 2:00AM New License
A license for another year was installed.

05/26/2011 2:00AM New SSL Cert and Oracle CPU
Oracle was updated to the latest CPU. A new SSL cert was installed. This new cert is using the NAU global sign CA. 

April 28, 2011
Rolling cluster restart
Each node in the cluster was restarted in turn to clear up some blocking locks in the database. This should not have affected end users.

April 25, 2011
3:50PM LMSfeed 1.7.1 deployed
A new version of the integration feed was deployed. This shouldn't change any Vista behavior.

April 14, 2011
2:00AM - 6:00AM Scheduled Maintenance
The memory_max_target on the production database for LEARPRD was decreased by 1GB to alleviate the excessive swapping that was happening on Bryce. This should improve performance.

April 7, 2011
12:00PM LMSfeed 1.7 deployed
A new version of the integration feed was deployed. This shouldn't change any Vista behavior.

April 5, 2011
7:00PM prd6 restart
prd6 was doing full GC's, it was restarted. This should not have affected end users.

March 24, 2011
2:00AM - 6:00AM Scheduled Maintenance
The cluster was restarted to allow the new kernel to be used.March 11, 2011

2:28PM prd2 reboot
Prd2 was accumulating webserver logs (webserver.logxxx) every 20 mintues.  Was restarted in an effort to clear the problem.  End users were not effected.

March 03, 2011
2:00AM - 6:00AM Scheduled Maintenance
The cluster was restarted to allow the new kernel to be used.

February 28, 2011
RedHat Patches Applied
No downtime. All updates from RHN have been applied to the Vista cluster. New kernel will be used next reboot.

February 17, 2011
2:00AM - 6:00AM Scheduled Maintenance
Oracle January CPU applied.

February 3, 20112:00AM - 6:00AM Scheduled Maintenance
Patches were applied to Vista to resolve a chat issue and to fix some potential security problems. Patches for VST-5274 and VST-5357 were applied and the cluster was restarted.

December 14, 2010
8:45AM prd3 reboot
Prd3 was restarted to clear a stuck job that was causing performance issues. This should not have affected end users.

November 16, 2010
10:52AM prd3 reboot
Prd3 had maxed out the JVM heap and was doing constant full garbage collections which made the node unresponsive. It was restarted and came back. This should not have affected end users.

October 31, 2010
9:15PM - 10:40PM
Vista was down due to disk array hardware issues.  A controller on the MD3000i disk array that hosts the Vista application node operating systems died.  The redundant controller was still functional, but all services needed to be rebooted to fix the error state they were in.

October 21, 2010
2:00AM - 6:00AM Scheduled Maintenance
Vista was updated to service pack 4 (8.0.4)

October 20, 2010
9:45AM prd3 reboot
Prd3 had maxed out the JVM heap and was doing constant full garbage collections which made the node unresponsive. It was restarted and came back. This should not have affected end users.

October 18, 2010
Search Indexing job change
The search indexing background job was changed from running hourly to running daily at 12:00AM to help reduce daytime load on the production database.

October 18, 2010
10:55AM prd5 reboot
Prd5 had maxed out the JVM heap and was doing constant full garbage collections which made the node unresponsive. It was restarted and came back. This should not have affected end users.

October 14, 2010
3:15PM prd4 reboot
Prd4 had maxed out the JVM heap and was doing constant full garbage collections which made the node unresponsive. It was restarted and came back. This should not have affected end users.

September 21, 2010
9:50PM prd3 reboot
Prd3 had maxed out the JVM heap and was doing constant full garbage collections which made the node unresponsive. It was restarted and came back. This should not have affected end users.

September 15, 2010
1:10PM prd4 reboot
Prd4 had maxed out the JVM heap and was doing constant full garbage collections which made the node unresponsive. It was restarted and came back. This should not have affected end users.

September 15, 2010
12:30PM Configuration Change
Updated the "Time of day to activate search index optimization" to 1:00AM from 5:00AM

September 14, 2010
9:50PM prd6 reboot
Prd6 was running the search indexing job which had started to put too much load on the database.  Prd6 was restarted to try to clear the job and the load on Bryce went back to normal.

September 13, 2010
3:30AM prd5 reboot
Prd5 had maxed out the JVM heap and was doing constant full garbage collections which made the node unresponsive. It was restarted and came back. This should not have affected end users.

September 7, 2010
10:20PM prd4 reboot
Prd4 was running the search indexing job which had started to put too much load on the database.  Prd4 was restarted to try to clear the job.  Eventually, the Oracle process that was running this job had to be killed in order to bring the load on Bryce back to normal.

September 1, 2010
2:28PM Change Tracking Background Job Schedule
The tracking background job was changed to start at 1:00AM instead of 4:00AM to alleviate some timing issues with the ETL job for vista_trk.

August 26, 2010
2:00AM - 6:00AM Scheduled Maintenance
/dbsave was moved back to the newly reconfigured volume and vista was brought back online.

August 19, 2010
2:00AM - 6:00AM Scheduled Maintenance
Bryce's disk array was reconfigured slightly. The /dbsave area was moved to a temp location so that a drive could be removed from the volume to create a hot spare for /db02. This volume will be rebuit throughout the week and /dbsave will be moved back to the new volume next week. Also, /db03 was resized since it was nearing capacity.

August 5, 2010
2:00AM - 6:00AM Scheduled Maintenance
Oracle CPU for July was applied to the production database.  New Vista java applets were installed to update the signing certificate for another year.

July 29, 2010
9:00AM prd5 reboot
Prd5 had maxed out the JVM heap and was doing constant full garbage collections which made the node unresponsive. It was restarted and came back. This should not have affected end users.

July 29, 2010
prd2 moved to VM cluster
Prd2 was moved to the new Xen virtual machine hardware cluster.

July 28, 2010
prd3 moved to VM cluster
Prd3 was moved to the new Xen virtual machine hardware cluster.

July 22, 2010
7:30AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

July 20, 2010
12:15PM prd4 reboot
Prd4 had maxed out the JVM heap and was doing constant full garbage collections which made the node unresponsive. It was restarted and came back. This should not have affected end users.

July 19, 2010
8:10AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

July 8, 2010
prd4 & prd5 moved to VM cluster
Prd4 & prd5 were moved to the new Xen virtual machine hardware cluster.

July 7, 2010
prd6 moved to VM cluster
Prd6 was moved to the new Xen virtual machine hardware cluster.

July 7, 2010
7:40AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

July 2, 2010
2:00PM Course Lifecycle Update
The VistaCourseLifecycle job was updated to fix a few bugs and to incorporate the new destination table that is now live in the override database.

July 1, 2010
2:00AM - 6:00AM Scheduled Maintenance
The Vista license key file was updated for 2010-2011. All servers were restarted to allow the new file to take affect.

June 29, 2010
2:30PM prd2 reboot
Prd2 was throwing errors when logging into CAS, restarted.

June 28, 2010
7:30AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

June 25, 2010
7:30AM prd3 reboot
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

June 22, 2010
7:50AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

June 18, 2010
7:40AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

June 17, 2010
7:25AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

May 23, 2010
7:45PM prd2 reboot
Prd2 was bounced as it was experiencing problems with the thick OCI Oracle connections. Perl jobs were stalled.  This should not have affected end users.

May 13, 2010
2:00AM - 6:00AM Scheduled Maintenance
Vista was shutdown for several maintenance items to be performed. The Oracle CPU for April 2010 was applied to Bryce. Batteries were replaced the in Bryce storage array. A change was made to the $VISTA_HOME/serverconfs/security/methods.xml file to remedy a security vulnerability reported by Blackboard.

was changed to

May 10, 2010
7:45AM prd2 reboot
Prd2 was restarted as it was experiencing periodic load issues.  This should not have affected end users.

April 28, 2010
8:05AM prd4 reboot
Prd4 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 27, 2010
8:33AM - 11:12AM prd3 hardware swap
A problem with the raid controller was causing inconsistent performance on Prd3.  Prd3 was swapped out with the old Denali.  End users who connected or were previous connected to Prd3 may have been effected.

April 26, 2010
7:25AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 24, 2010
7:20AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 22, 2010
7:35AM prd5 reboot
Prd5 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 22, 2010
3:30AM prd6 reboot
Prd6 was rebooted during the maintenance window.  Prd5 was accumulating users, and in a state of late might/early morning delirium, prd6 was rebooted by mistake. This did not affect end users.

April 16, 2010
8:50AM LMSfeed 1.5.2 deployed
A new bugfix version of the LMSfeed was deployed to production. This fixes a case where some enrollment records weren't being updated in the override database.

April 16, 2010
8:15AM prd3 reboot
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 16, 2010
7:20AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 14, 2010
7:20AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 13, 2010
7:30AM prd3 reboot
Prd3 was experiencing load warnings several times daily, as well as a sustained moderate load, thus is was restarted. This should not have affected end users.

April 12, 2010
7:40AM prd2 reboot
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 2, 2010
LMSfeed 1.5 deployed
The new version of the LMSfeed, 1.5, was deployed to tcat1 and is now processing.

March 29, 2010
8:30AM prd6 reboot
Prd6 was accumulating users, it was restarted to clear them out. This should not have affected end users.

March 24, 2010
7:30AM prd4 reboot
Prd4 was rebooted to clear up an issue with remote storage. This should not have affected end users.

March 23, 2010
1:15PM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

March 18, 2010
2:00AM - 6:00AM regular maintenance
Bryce.vista.nau.edu was changed out for new hardware.

March 11, 2010
9:45AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

March 9, 2010
7:45AM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

March 8, 2010
8:45AM prd4 restart
Prd4 was accumulating users, it was restarted to clear them out. This should not have affected end users.

March 4, 2010
3:30PM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

February 25, 2010
2:00AM - 6:00AM Scheduled maintenance
Vista was moved from behind the old CSM load balancer to the ACE load balancer.

February 23, 2010
7:40AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

February 19, 2010
7:45AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

February 16, 2010
1:45PM - 2:00PM Vista Service Interruption
Power was lost to one of the networking switches that services the Vista cluster. Power was restored and Vista service was returned.

February 15, 2010
7:55AM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

February 12, 2010
7:30AM prd4 restart
Prd4 was accumulating users, it was restarted to clear them out. This should not have affected end users.

February 10, 2010
7:10AM prd5 restart
Prd5 was accumulating users, it was restarted to clear them out. This should not have affected end users.

February 8, 2010
7:30AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

February 3, 2010
7:40AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

January 29, 2010
3:30PM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

January 28, 2010
7:30AM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

January 28, 2010
9:45PM prd6, prd5 restart
Prd5 was accumulating users, it was restarted to clear them out. Prd6 had some runaway processes that were loading up the host and causing that node to be unresponsive, it was restarted to fix. This should not have affected end users.

January 26, 2010
7:45AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

January 21, 2009
8:45AM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

January 17, 2009
6:30AM - 7:45AM Vista cluster maintenance
A restart was scheduled and completed due to the CAS system upgrade.

December 23, 2009
8:55AM prd2 restart
Prd2 was restarted in an attempt to clear a lock held by a session running on Prd2.

December 15-16, 2009
8:50PM - 7:30 prd4 crash
The prd4.vista.nau.edu machine crashed for an unknown reason. This brought down the prd4 and prd5 vista nodes. The server was rebooted and both application nodes were brought back online. This should not have affected end users.

December 15, 2009
10:30AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

December 14, 2009
1:28PM - 2:18PM Vista outage
A fibre channel switch rebooted in the ITS machine room which caused the Vista backend database to go down for the duration of this outage. Once the connections were restored, the database was brought back online and Vista was returned to service.

December 10, 2009
11:41AM prd4 crash
Prd4 crashed from a memory error. All users should have found a new home on another node. This should not have affected end users.

December 9, 2009
8:00AM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

December 6, 2009
12:20PM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

December 3, 2009
2:00AM - 6:00AM Regular Maintenance
The October Oracle critical patch update was applied to bryce.

November 30, 2009
7:35AM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

November 25, 2009
7:45AM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

November 19, 2009
9:20PM - 10:10PM Vista database down
Due to campus-wide power failure, excessive heat in the ITS machine room caused a fibre switch to reset which caused Vista's database to go down. This made Vista unavailble until the database was brought back online. The Vista mid-tier stayed up and reconnected to the Oracle DB once it was running again.

November 18, 2009
New course lifecycle process
The latest iteration of the course lifecycle code was put into production. This new version now exports gradebooks before deleting a section with content and also sends out section info emails to instructors a week before their sections start. The cron was also changed to run these jobs at 9AM instead of 1AM to be sure that the data warehouse isn't being refreshed while they run.

November 18, 2009
10:30AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

November 17, 2009
7:15AM - 7:40AM prd4, prd5, prd6 restarted
Prd5 was accumulating users and prd6 was having system load issues so they were restarted to clear these issues. Prd was inadvertantly restarted.

November 9, 2009
8:45AM prd2, prd4, prd5 restart
Prd2 and prd5 were accumulating users, they were restarted to clear them out. This should not have affected end users.  Prd4 was also inadvertantly restarted.

November 3, 2009
7:15AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.  The webct Oracle password for DB Bryce was updated on nodes prd2, prd3, prd4 and prd5 in "startup.properties".

October 26, 2009
10:30PM prd4 restarted
Prd4 ran out of available JVM heap space and became unresponsive. It was restarted to bring it back into service. End users should not have been affected.

October 15, 2009
2:00AM - 6:00AM Regular Maintenance
Vista was restarted, a patch was put in place to fix a quiz bug that was resulting in blank answers. Prd6 was placed into service on the build1.vista.nau.edu machine. The webct Oracle password on Bryce was changed.

October 13, 2009
10:21AM prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

October 12, 2009
8:30AM prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

October 6, 2009
10:30AM Database cleanup
Ran the following script to clean up 33 orphaned cms_content_entry records

October 1, 2009
2:00am full restart
The JDBC driver was updated to 10.2.0.3.0. The old version lives at /export/vista/vista4/ojdbc/ojdbc14.jar.old

September 28, 2009
7:10am prd2 restart
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

September 24, 2009
9:00am prd3 restart
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

September 17, 2009
8:00am prd4/5 restarted
After the full restart in the AM prd3,4 and 5 were not mounting /export/vista. Another restart was preformed on prd4/5 to see if that would clear up the problem. Users should not have been effected. The reboot did not solve the issue. I ended up turning off autofs, and modifing /etc/fstab to include the IP of royal and that solved the problem.

September 16, 2009
4:00pm prd3 restarted
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

September 10, 2009
3:45pm prd3 restarted
Prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

September 1, 2009
3:45pm prd2 restarted
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

August 31, 2009
8:15am prd2 restarted
Prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

August 28, 2009
1:00PM LMSfeed 1.4 deployed
The latest revision of the LMS feed was deployed to production and started up.

August 28, 2009
11:00am prd3 restarted
Prd3 was restarted because it had exhausted its free JVM heap space. End users should not have been affected.

August 25, 2009
12:00PM LMSfeed 1.3 deployed
The latest revision of the LMS feed was deployed to production and started up.

August 24, 2009
9:05AM Login page change
The Vista login page was changed to back to use CAS single sign-on. This change did not affect already logged in Vista users.

August 24, 2009
10:55AM Login page change
The Vista login page was changed to bypass CAS while a new CAS server is brought up to handle the extra load. This change did not affect already logged in Vista users.

August 23, 2009
8:30AM - 11:00AM
A campus power outage caused issues with the production Vista database. Vista was down until the issues were resolved and the database was brought back online.

August 13, 2009
2:00AM - 6:00AM Regular Maintenance
Vista was upgraded to the latest release, 8.0.3.

August 6, 2009
2:00AM - 6:00AM Regular Maintenance
The Vista production was patched with VST 3533 client patch. See case id 609003. Short story, restores now work. Prd3 is still down due to some unrelated issue.

July 29, 2009
2:00AM - 6:00AM Regular Maintenance
The Vista production database was patched with OCP for April. The Wimba powerlink was upgraded to version 4.0.0.

July 29, 2009
10:15PM Login page changed
Added an extra warning on the login page to remind users of the weekly maintenance window. This should only appear after noon on Wednesdays and until the 2:00AM maintenance window.

July 29, 2009
1:40 PM prd3 restarted
prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

July 26, 2009
4:00AM - 9:00AM Scheduled Maintenance
Power was turned off in the ITS machine room for electrical work. Vista was unavailable during this time.

July 21, 2009
10:20 AM prd3 restarted
prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

July 2, 2009
2:00AM - 6:00AM Regular Maintenance
A new license file good through June 30, 2010 was installed on all the nodes.
The old Elluminate 1.104 proxy tool was cleaned up after a failed deletion attempt via scripts provided by Bb, Delete a proxy tool that fails to delete from the GUI

June 26, 2009
1:10 PM prd3 restarted
prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

June 23, 2009
Course Lifecycle code updated
The "Destroyer" code that deletes old sections and sends out notifications was updated to fix a couple of bugs around chair emails SVN 1494

June 20, 2009
4:15PM prd3 restarted
prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

June 17, 2009
3:15PM course lifecycle job updated
A new version of the course lifecycle job was deployed to tcat1. This updated fixed a problem that occured when the vista_ims_adm schema was changed for the new feed and also adds in a check to not delete a section as long as there is an open eval for it.

June 15, 2009
3:15PM prd2 restarted
prd2 was rebooted to attempt to clear out a networking issue, no users should have been affect by this.

June 11, 2009
7:20AM prd2 restarted
prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

June 4, 2009
Deployed Annotate proxy tool
The Annotate proxy tool was created based on the generic outgoing SSO proxy tool. This points at acadtools.ucc.nau.edu/annotate which was developed by ELC.

June 4, 2009
Oracle CPU and ePack fix
DBA's upgraded to the latest Oracle CPU. Fixed an ePack issue with 1094-1512 that required a restart, did this while down. See DBA's log for more info on CPU.

June 2, 2009
Feed updated to LMSfeed1.2
New feed code was put into place to fix problems with zero-credit sections in the override database.

May 31, 2009
12:00PM prd2 restarted
prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.

May 30, 2009
Feed updated to LMSfeed1.1
New feed code was put into place to fix performance, content section, and expire_ts issues.

May 28, 2009
4:20PM - 4:45PM Full Cluster Restart
While applying a new SSL cert the WebLogic console tried to migrate JMS to another server. This was not successfull and killed the cluster. A full cluster restart was needed. The outage should not have been more then 20 mins.

May 27, 2009
New feed!!
A new LMS integration feed was put into place.

May 21, 2009
9:30AM - 10:30AM Restart prd2 and prd5
prd2 and prd5 were accumulating users. The servers were restarted one at a time and users were given time to move to another server. This should not have affected end users.

May 12, 2009
5:20PM - 5:25PM Bb Vista down
The disk array that houses bryces VISTA4 Oracle database had a LUN failure which caused
/db01 to lose access to its data. This was due to the RAID 5 losing two disks.  One had failed
over the weekend and as Clint was about to swap it another failed.  Clint was able to revive
the second failed disk to get the LUN back and swapped the initial failed disk.  Thus, Vista
never failed, it just lost the ability to talk to the back end.

May 11, 2009
8:45PM prd3 restarted
prd3 died due to running out of heap space and was restarted. This should not have affected end users.

May 4, 2009
7:00AM prd4 restarted
prd4 was accumulating users, it was restarted to clear them out. This should not have affected end users.

May 2, 2009
7:20AM prd3 restarted
prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 29, 2009
7:30AM prd3 restarted
prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 28, 2009
3:00PM prd2 restarted, 3:15PM prd4 restarted
prd2 was accumulating users, it was restarted to clear them out. This should not have affected end users.
prd4 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 17, 2009
3:15PM prd4 restarted
prd4 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 13, 2009
3:15PM prd3 restarted
prd3 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 13, 2009
7:20AM prd5 restarted
prd5 was accumulating users, it was restarted to clear them out. This should not have affected end users.

April 5, 2009
12:30PM prd3 restarted
prd3 started with having a high load and then started accumulating users, it was restarted to clear them out. This should not have affected end users.

April 2, 2009
11:20AM prd3 restarted
prd3 started with having a high load and then started accumulating users, it was restarted to clear them out. This should not have affected end users.

March 29, 2009
1:20PM prd2 restarted
prd2 started with having a high load and then started accumulating users, it was restarted to clear them out. This should not have affected end users.

March 26, 2009
7:15AM prd3 restarted
prd3 started with having a high load and then started accumulating users, it was restarted to clear them out. This should not have affected end users.

March 25, 2009
2:45PM TCP Settings Changed

Setting

Old Value

New Value

/proc/sys/net/core/rmem_default

65535

252143

/proc/sys/net/core/rmem_max

131071

252143

/proc/sys/net/core/wmem_default

65535

252143

/proc/sys/net/core/wmem_max

131071

252143

/proc/sys/net/ipv4/tcp_timestamps

1

0

/proc/sys/net/ipv4/tcp_sack

1

1

/proc/sys/net/ipv4/tcp_window_scaling

1

1

The following was added to rc.local on each application node

March 20, 2009
2:15PM prd3 restarted
prd3 started with having a high load (7:45AM) and then started accumulating users, it was restarted to clear them out. This should not have affected end users.
March 5, 2009
2:00AM - 6:00AM Regular Maintenance
The Vista cluster and production database was restarted to allow a database configration change to take effect. init parameter "compatible" was changed from 10.2.0.3 to 10.2.0.4.
February 24, 2009
9:45PM prd2 restarted
Prd2 died and needed to be restarted. End users should not have been affected.

February 24, 2009
8:25AM prd2 restarted
Prd2 had elevated load and was doing some full GC's that were taking several minutes so it was restarted. End users should not have been affected.

February 22, 2009
3:20PM prd3 & prd4 restarted
Prd3 and prd4 died unexpectedly and were restarted. This should not have affected end users.

February 19, 2009
2:00AM - 6:00AM Regular Maintenance
Two java classes (GenericDiscussionDao.class & GenericRosterDAO.class) were replaced in the webct.ear file in order to fix errors. This is supposed to also fix the problem that was causing periodic Vista node crashes due to low free heap space. A copy of the original webct.ear file was made $VISTA_HOME/webctapp/webct.ear.backup.20090218, if this patch causes any new problems, it can be reversed by reverting to this copy of the file.

February 18, 2009
7:15AM prd4 restarted
prd4 was accumulating users, it was restarted to clear them out. This should not have affected end users.

February 12, 2009
8:15AM prd3 & prd4 restarted
prd3 and prd4 had an elevated load. Prd3 was restarted to clear the problem, but the restart had no effect. It was determined that the chat server was the issue and it was restarted on prd3 and prd4 to clear the elevated load.

February 6, 2009
8:30AM prd3 restarted
prd3 was accumulating users (>2000) and needed to be restarted.

February 2, 2009
Course Lifecycle Update
The course lifecycle tool was updated to remove lone content sections that are past the 106 week life time. SVN: 1380

January 27, 2009
2:15PM prd5 restarted
prd5 was doing full GC's and needed to be restarted.

January 23, 2009
2:15PM prd5 restarted
prd5 was accumulating users and needed to be restarted.

January 20, 2009
6:15PM prd4 restarted
prd4's free JVM heap space bottomed out and caused full garbage collections. It needed to be restarted as a result.

January 20, 2009
7:30AM prd2 restarted
prd2 was accumulating users and needed to be restarted.

January 12, 2009
7:15AM prd3 restarted
prd3 was accumulating users and needed to be restarted.

January 2, 2009
7:00AM - 11:05 PM
Vista was offline for 16 hours for power switch over and machine room realignment.

December 22, 2008
8:30PM prd2 restart
prd2 was accumulating user sessions and experiencing a heavy load.  It was thus restarted.

December 12, 2008
7:45AM prd2 restarted
the load on prd2.vista.nau.edu had been elevated for awhile, so Vista node prd2 was restarted. This brought the load back to normal.

December 5, 2008
7:30AM prd2 restarted
the load on prd2.vista.nau.edu had been elevated for about 36 hours, so Vista node prd2 was restarted. This brought the load back to normal.

December 2, 2008
3:15PM prd2 restart
prd2 was accumulating user sessions and was restarted.

November 27, 2008
2:00AM - 6:00AM Regular Maintenance
The cluster was restarted and the new Wimba powerlink was deployed (v3.3.1)

November 25, 2008
7:30AM prd3 restart
prd3 was accumulating user sessions and was restarted.

November 21, 2008
8:00AM prd5 restarted
prd5.vista.nau.edu was accumulating users (over 3000!), so Vista node prd5 was restarted.  This brought the number of users on prd5 back into sync with the rest of the nodes.

November 19, 2008
8:00AM prd2 restarted
the load on prd2.vista.nau.edu was slightly elevated, so Vista node prd2 was restarted. This brought the load back to normal.

November 13, 2008
2:00AM
7:30AM prd2 restarted
the load on prd2.vista.nau.edu was very elevated, (about 10 times the load on the other nodes), so Vista node prd2 was restarted. This brought the load back to normal.

November 11, 2008
7:20AM prd4 restarted
the load on prd4.vista.nau.edu was slightly elevated, so Vista node prd4 was restarted. This brought the load back to normal.

October 10, 2008
Changes to backup policy
Documented here: https://espweb.ucc.nau.edu/twiki/bin/view/DBA/VistaChangeLog

GC Fix scripts ran
2 scrips were run on bryce to fix small amounts of corruption that were causing GC to fail.

October 9, 2008
5:15PM - 6:05PM Vista Crash
Around 5:15PM prd1 went on a rampage, pegging its cpu and also peggin bryce's cpu such that neither were responding. Prd1 was power cycled and the load on bryce came down to normal. Vista was restarted at that point and everything was ok.

September 29, 2008
prd5 restart
prd5 was accumulating user sessions and was restarted.

September 19, 2008
Purge criteria update for Vista user accounts
primary affiliation "previous faculty" has been added to the list of affiliations which will incur an account purge, if the user account is NOT attached to any for-credit courses.  (EJS)

September 18, 2008
07:44:52
VISTA4@chaco.vista.nau.edu: alter system set db_block_checking=TRUE scope=both;
System altered.

September 15, 2008
8:30 AM prd4 crash
prd4 crashed due to jvm free heap space crash. This issue was supposedly fixed by upping the permgen memory limit to 256MB, but evidently did not. Bb ticket was re-opened about this issue.

August 29, 2008
permgen parameter change
The customconfig/startup.properties file was updated to set the permgen max space to 256MB to prevent the JVM free space from bottoming out. It should take effect the next time the nodes are restarted.

August 26, 2008
Db parameter changes
VISTA4@bryce.vista.nau.edu: alter system set undo_retention=86400 scope=both; System altered.

VISTA4@bryce.vista.nau.edu: alter system set optimizer_index_caching=90 scope=both; System altered.

VISTA4@bryce.vista.nau.edu: alter system set optimizer_index_cost_adj=20 scope=both; System altered.

VISTA4@bryce.vista.nau.edu: !date
Mon Aug 25 14:54:55 MST 2008

August 21, 2008
2:00AM - 4:00AM Regular Maintenance - Java cert update
-The ear file was updated with a new java cert
-Vista was restarted
-The netbackup client on Bryce was updated to a newer version

August 14, 2008
2:00AM - 4:00AM Regular Maintenance - Vista 8.0.1 upgrade
Vista was upgraded from 4.2.3 to 8.0.1

August 12, 2008 
Bill Loucks: crontab on bryce:
Stop doing disk backups and switch tape full backup to Saturday 1-AM time:

REM Full (incr level 0) RMAN Hot backup to TAPE Sat 1:00
00 01 * * 6 . ./.bash_profile >/dev/null 2>&1; oburmantape VISTA4 0 y Oracle-OBackup Backup-3m

REM Incr (incr level 1) RMAN Hot backup to TAPE Mon Tue Fri 1:00
00 01 * * 1,2,5 . ./.bash_profile >/dev/null 2>&1; oburmantape VISTA4 1 y Oracle-OBackup Backup-3m

August 12, 1008 
7:20AM prd3 restart
prd3 was accumulating users so it was restarted to prevent any issues.

August 8, 2008 
7:35AM prd3 restart
Restarted prd3 to clear stuck search indexing job and inserted record to prevent search indexing until issue can be resolved.

August 7, 2008 
2:00AM - 6:00AM Regular Maintenance
Oracle database was upgraded to 10.2.0.4

July 31, 2008 
2:00AM - 6:00AM Regular Maintenance
Vista production cluster was cycled.  Oracle hotspots, tablespaces "webct_data" and "webct_index," were moved over to other disks in hopes of boosting preformance during times of high system demand.  "webct_data" was moved to "db02" and "webct_index" was moved to "db03".

July 24, 2008 
2:00AM - 6:00AM Regular Maintenance
Vista production cluster was cycled.  PRD1 was rebooted, as Oracle connectivity on that node had been lost.

July 21, 2008 
7:40AM Prd3 restart
prd3 was accumulating sessions, over 500.  No end users should have been impacted from the restart. (EJS)

July 3, 2008 
2:00AM - 6:00AM Regular Maintenance
A new license file for 08-09 was installed, Vista was restarted and the LDAP configuration was changed back to ldap.nau.edu.

July 2, 2008 
9:49AM Prd3 restart
prd3 was accumulating sessions, over 900.  No end users should have been impacted from the restart.

June 20, 2008
Updated PS2DB.jar
The PS2DB.jar file was updated so that the 2 db feed will now check to see if a section has been deleted and will not feed it into the override database if it has.

2008-06-11
Database vista4.acadia.vista.nau.edu change of parameter aq_tm_processes
alter system REset aq_tm_processes scope=spfile sid='*'; # Autotune, but requires database bounce
alter system set aq_tm_processes = 1; # To set a nonzero value in the running instance

June 2, 2008
1:45PM Prd3 restart and disabled search indexing
Restarted prd3 to clear oracle 04030 errors. Search indexing was disabled to prevent this from happening until the problem can be resolved. Search indexing was disabled by inserting this record into the Vista DB.






June 2, 2008
8:15AM Prd3 restart
prd3 was causing a large number of ORA-04030 errors.  A restart of prd3 seems to have cleared the issue.  No end users should have been impacted from the restart.


















May 29, 2008
2:00AM - 6:00AM Regular Maintenance
The vista cluster including database hosts were shutdown in order to replace the backup batteries in the Vista SAN's.  SSL Certificates were also updated.

May 22, 2008
12:00PM - OS Restart of prd2
The operating system needed a restart to fix an issue with perl DBI not being able to connect to the Vista database.

May 22, 2008
7:00AM - Restarted prd2
prd2 was restarted to clear a bunch of slowly running or hung processes.  There were a lot of feeds, both peoplesoft and LDAP that were all running simultaneously and had been for many hours.  This was causing prd2 to be bogged down and the jobs weren't completing. 

May 16, 2008
4:00PM - changed search indexing to daily
search indexing was switched from an hourly run schedule to a daily run schedule. It's set to kick off at 2:00AM each day.

May 16, 2008
9:30AM - Prd5.vista.nau.edu died
Prd5.vista.nau.edu suddenly ran out of free JVM heap space and began doing nonstop full GC's. It had to be restarted to bring service back to the node. No users should have been affected.

May 14, 2008
Full DB backup was prematurely killed
Wednesday mornings full DB backup was accidently killed around mid-morning, near its completion. (EJS)

2008-05-14 14:20 Bill Loucks
Made some changes to the "oburmantape" script:

1. The "oburmantape" script no longer tries 4 times; it does only one pass.

2. Instead of a single rman command which says basically "backup database plus archivelog", it is three separate rman sessions, as follows: (1) backup archivelog all, (2) backup database, and (3) backup archivelog all.

3. Instead of the controlfile autobackups being configured at the rman level (resulting in the controlfile autobackup being performed whenever any type of backup is done), I've deconfigured controlfile autobackup at the rman level, and the "oburmantape" script does one last separate rman session as follows: backup current controlfile.

May 8, 2008
7:45AM - Prd1 removed from LB pool
prd1 was removed from the LB pool by runnint /root/startDowntime.sh to prevent users from being sent to it in order to prevent the accumulation of users sessions which has been happening periodically.  This will hopefully prevent the need to restart prd1 during normal operations.  This will remain through the summer unless something changes and a plan for adding an additional node will be needed before the start of fall.

May 7, 2008
4:10PM - Unstable Vista performance among nodes
At 3:52PM, HAL ticket #18009 reported inconsistent behavior among Vista nodes in displaying a Final Exam for EE220:  prd3 and prd5 displayed the exam, prd2 and prd4 did not.  At 4:10PM, prd1 was added back into the LB pool (./stopDowntime.sh), as it was removed earlier in the day due to accumulated sessions on that node.  At the same time, both prd2 and prd4 were taken out of the LB pool (./startDowntime.sh) and prd2 was cycled and back up at 4:34PM.  It was then added back into the LB pool.  prd4 remained out of the LB pool as the JMS had migrated to it earlier in the day.  (EJS)

May 7, 2008
5:00AM - Prd4 died unexpectedly
Prd4 was throwing "Connection administratively disabled" messages after losing contact with the JMS server and shut down, it was restarted.

May 6, 2008
2:30 PM - Prd2 chat issues
The chat server on prd2 lost it's connection to the JMS server and wouldn't recover, after a restart, we disabled the chat service and brough the server back up. It should be re-enabled at the next scheduled maintenance period and the cluster restarted.

April 10, 2008
2:00AM-6:00AM - Scheduled Maintenance

  • A patch was installed fix a caching bug. This fix may improve performance especially for users with dialup or slow broadband connections.
  • Some parameter changes were made to the who's online database tables to fix some errors that were appearing there.
  • The Vista cluster was restarted.

2008-03-21 13:12 - Bill Loucks
In Vista Production (vista4.bryce.vista.nau.edu)

2008-03-18 10:58 - Bill Loucks
In Vista Production (vista4.bryce.vista.nau.edu)

  • Changed parameter statistics_level from 'ALL' to 'TYPICAL' with: alter system set statistics_level=ALL scope=both;

March 13, 2008
2:00AM-6:00AM - Scheduled Maintenance
The database server that is used by Vista was upgraded from version 9i to 10g.

March 6, 2008
2:00AM-6:00AM - Scheduled Maintenance
The database server that is used by Vista had it's operating system upgraded from Red Hat EL, AS3 to AS4.

February 21, 2008
2:00AM-6:00AM - Scheduled Maintenance
A change was made to restrict background jobs from running on prd1, the primary application server, this will allow us to kill jobs that are affecting performance without affecting end users. The cluster was also restarted.

February 14, 2008
2:00AM-6:00AM - Scheduled Maintenance
Vista was successfully upgraded to version 4.2.3.

January 31, 2008
2:00AM-6:00AM - Scheduled Maintenance
Changed the XX:MaxPermSize JVM param from 192m to 256m. Updated the scripts on prd4 to also take prd5 out of service during the outage window.

January 17, 2008
5:45PM - 6:30PM - Unexpected Outage
A problem with the Vista load balancer was causing all users to timeout when attempting to access vista. The load balancer was failed over to the backup and service was restored.

January 17, 2008
2:00AM - 6:00AM Regular Maintenance
Configuration changes were made to help increase the stability of the cluster, a new Wimba powerlink was installed and the cluster was restarted.

January 16, 2008
6:30PM - Unexpected Outage
Prd4 was restarted to clear up a replication issue. For an unknown reason, this caused the rest of the application nodes to shutdown. The cluster was restarted and service was restored.

January 11, 2008
4:00PM - Added Prd5 Vista Node
A fifth application node was added to the Vista cluster in an attempt to increase system stability.

January 9, 2008
1:30PM - 1:40PM - Unexpected Outage
Vista was experiencing severe performance and stability issues. This required a restart of the system to clear the problems.

December 20, 2007
2:00AM-6:00AM - Regular Maintenance
During this scheduled maintenance, we installed the scholar powerlink, fixed chat/whiteboard SSL certificates and installed another application node.

December 19, 2007
4:00PM - 4:20PM Unexpected outage
The Vista servers were rebooted to alleviate various chat connection problems and odd behaviors within Vista.

December 6, 2007
2:30AM-3:00AM - Regular Maintenance
The Vista servers were rebooted to enable use of version 1.5 of the Generic Outgoing SSO PowerLink for use with the IITV Video Project and the initial connections for the WEBCT$PF user was set to 1 per node.

November 16, 2007
5:40AM - 6:10AM - Unexpected Outage
The Vista servers had to be rebooted this morning to clear an out of memory condition which would have crashed them within a few hours.

November 15, 2007
2:00AM - 6:00AM - Vista 4.2.2 upgrade
Vista was upgraded to the latest release, version 4.2.2 which fixes a number persistent issues including fixes for office 2007 file downloads.

November 8, 2007
2:00AM - 4:00AM - Regular Maintenance
The Vista cluster was restarted during the scheduled maintenance window.

November 2, 2007
5:45AM - 7:25AM -Unexpected Outage
The first application node experienced an operating system failure. This failure resulted in the subsequent outage of the remaining Vista servers. Service was restored as soon as the first node was brought back online.

October 28-29, 2007
- TurnItIn powerlink not functioning
There was a licensing problem with the TurnItIn powerlink that caused the service to unavailable all day on the 28th until 8:00AM on the 29th. Once techsupport at TurnItIn was able to be contacted, they fixed their issue and service was restored. No TurnItIn assignments could be submitted during this time.

October 22, 2007
1:00PM - 2:00PM - Unexpected Outage
Prd1 of the Vista cluster experienced a low memory condition. A rolling restart of the Vista cluster failed which caused a full shutdown/restart to be nessecary.

October 04, 2007
2:00AM - 6:00AM - Regular Maintenance
During this maintenance, logging was changed to provide better reporting.

September 27, 2007
2:00AM - 6:00AM - Regular Maintenance
During this maintenance, we installed a 4th application node for the Vista cluster in order to increase capacity due to growth that we have experienced this fall.

September 16, 2007
3:00AM - 9:30AM - Downtime due to power outage
Power was turned off to the ITS building to allow crews to fix a problem caused by the accidental outages from several weeks ago. Vista was down during this time and service was restored once power was turned back on to the ITS building.

September 12, 2007
1:30PM - 1:45PM - Unexpected outage
Prd1.vista.nau.edu became slow and unresponsive. An attempt was made to move all users to the 2 remaining Vista application servers, but doing so overloaded the 2 good nodes with too many users and Vista went down. An additional server was ordered prior to this event to increase capacity, it should be in place soon.

September 6, 2007
8:30AM - 9:00AM - Unexpected outage
A problem was discovered with the patch that was put into place earlier in the day which caused learning modules to become inaccessible. Vista was restarted in order to remove the patch.

September 6, 2007
2:00AM - 6:00AM - Regular maintenance
A patch was put in place to fix issues with special characters that were causing display issues.

August 26, 2007
5:00PM - 8:00PM - Outage due to power failure
Vista was taken down during this time to accommodate a planned power outage to restore permanent electrical service to central campus.

August 25, 2007
7:30AM - 11:50PM - Outage due to power failure
Vista was down all day due to several power outages across central campus.

August 23, 2007
2:00AM - 6:00AM - Regular Maintenance
Vista was restarted after a successful test of a rolling restart. The proxy authentication module was also installed.

August 19, 2007
2:00AM - 6:00AM
Vista was upgraded to the latest release, Vista 4.2.1.

August 17, 2007
12:00PM - 2:00PM
Vista was access was disabled for 2 hours so that ITS and Blackboard support could troubleshoot the issues that appeared during the upgrade attempt on the 16th.

August 16, 2007
2:00AM-7:00AM - Vista AP2 Upgrade Attempt
An upgrade to Vista 4.2.1 was attempted this morning. After successful upgrade, the system was unstable and the decision was made to revert to Vista 4.1.1

August 9, 2007
2:00AM - 6:00AM - Regular Maintenance
The Vista cluster was restarted and the fixOrganizerLinks maintenance script was run.

August 3, 2007
1:10PM - 1:20PM - Unexpected Outage
Prd1 experienced 100% cpu utilization and stopped responding which resulted in all application servers following suit. All servers were restarted and service was restored in approximately 10 minutes.

July 12, 2007
2*:00AM - 5:00AM - Planned Outage*
- The Vista cluster was taken down to install the new license key for the next year. This license will be valid until June 30 2008.

July 2, 2007
9:00AM - 10:30AM - Unexpected Outage
- The Vista application servers had been experiencing intermittent database connection problems which was causing students to be kicked out of exams or quizzes unexpectedly. The Vista cluster was restarted to alleviate this situation and return Vista to normal operation.

June 10, 2007
11:15AM - Prd3 Restart
*-*Prd3 experienced 100% cpu utilization and stopped responding. The node was restarted, no outage should have been observed by any users.

May 24, 2007
2:00AM - 6:00AM - Regular Maintenance
*-*Some JMS configuration changes were made in hopes to improve Vista's reliability.
-The vista.nau.edu SSL certificates were updated with renewed certificates.
-The cluster was then restarted.

May 22, 2007
11:00 AM - 11:30 AM - Unexpected Outage
Vista had to be restarted unexpectedly. This was caused by a complication from a network outage that had happened earlier in the day.

May 10, 2007
6:00 AM to 6:10 AM - System Downtime
Set CountOfRefreshFailuresTillDisable="0" and CountOfTestFailuresTillFlush="0" in config.xml. Origional values are both "1". This should allow us to re-enable the search index job.

May 9, 2007
8:35 AM - Prd2 Restart
-Restarted prd2 to clear the 100% cpu state that prd2 was in, no users should have been impacted. 100% cpu state began at 12:20AM, 5/09/07

April 28, 2007
9:00 PM - 10:00 PM - Prd3 Restart
-Prd3 died unexpectebly, it was restarted, no users should have been impacted.

April 25, 2007
4:15 PM - Unexpected Outage
-A java error on one of the Vista application nodes triggered instability with the cluster. We had to shutdown Vista and restart it to restore functionality. Vista was not functional for approximately 15 minutes.

April 23, 2007
9:45 AM - Prd3 Restart
-Restarted prd3 to clear the 100% cpu state that prd3 was in, no users should have been impacted.

April 19, 2007
2:00AM-3:00AM - Weekly Maintenance
- Updated Cas Authentication module
- Rotated Logs

April 10, 2007
9:45 AM - Prd3 Restart
-Restarted prd3 to fix an issue with assessments that was impacting instructors and students on prd3. Only prd3 should have been affected and users should not have observed any outage during the restart. The assessments should now be working correctly.

April 4, 2007
10:00 AM - Prd3 Shutdown
-Shutdown prd3 to fix intermittent connection problems, no users should have been impacted. Will leave prd3 down until the scheduled maintenance to happen early tomorrow morning.

April 3, 2007
10:15 PM - Prd3 Restarted
-Restarted prd3 to fix intermittent connection problems, no users should have been impacted.

March 16, 2007
7:30 AM - Prd2 Restarted
-Restarted prd2 to clear a database lock, no users should have been impacted.

March 8, 2007
2:00AM-3:00AM - Weekly Maintenance
- Updated License Key
- Set search index not to run until we get fix
- Rotated Logs

February 21, 2007
12:50PM - Unexpected outage
At around 12:50, prd1 lost its database connection which caused the Vista cluster to stop responding, it was restarted and full service was restored at around 1:15PM.

2:00AM - Weekly Maintenance
-Shutdown Vista cluster
-Ran webct.sh cleanLogs
-Ran createWeblogs.pl
-Started Vista cluster

February 20, 2007
8:00AM - Prd3 Database Connection Failed
Prd3 was restarted and the database connection was restored. No outage should have been observable by end-users.
2:00AM - Vista Cluster Restart
The Vista application cluster was restarted to help fix the stability problems seen over the past several days, the total downtime observable by end-users was around 10 minutes.

February 19, 2007
2:45PM - JMS Connection Failed on prd3
Prd3 was restarted and JMS was restored. No outage should have been observed by any end-users.
12:00PM - JMS Connection Failed on prd2
Prd2 was restarted and JMS was restored. No outage should have been observed by any end-users.

February 17, 2007
VIsta Mail and File Uploading problems
Mail within Vista was being returned as undeliverable and file uploading was not working during various parts of the afternoon. Everything was functioning properly after several rolling restarts of the system. Vista remained up and operational during these restarts.

Februrary 16, 2007
VIsta mail not working
Mail within vista was not working for most of the day, but began working around 3:00PM

February 15, 2007 2:00 AM - 2:40 AM
Vista system upgrade
Hotfix 2 successfully applied to Vista 4.1.1..

February 8, 2007 2:00 AM - 8:02 AM
Vista system upgrade
Vista was successfully upgraded from version 4.0.3 to version 4.1.1 w/ hotfix 1.

February 1, 2007 12:46 PM - 12:55 PM
Unexpected Outage
At around 12:46 PM all 3 of the Vista servers shut down due to an issue on prd1.vista.nau.edu. Service was restored in approximately 10 minutes.

February 1, 2007 02:00 AM - 05:06 AM
Regular Maintenance
1) Shutdown Vista
2) Ran verifyUniqueName_Oracle.sql
3) Ran fixUniqueName_Oracle.sql
4) Ran verifyOrgPgLayoutOracle.sql
5) Ran fixUniqNameMissingDiff_4_0_x_Oracle.sql
6) Ran sh fixOrganizerLinks.sh detect from admin2.vista.nau.edu
7) Ran sh fixOrganizerLinks.sh fix from admin2.vista.nau.edu
8) Ran sh fixOrganizerLinks.sh detect
9) Cleaned the logs on all 3 nodes
10) Run truncateXBTables.sql
11) Ran createWebLog.pl on all 3 nodes
12) Ran reszdata.sql
13) Resized 4 DB files
14) Ran fix_simple_file_unique_name.sql
15) Ran Fix500ErrorsScripts/SI117426_fix_all.sql
16) Ran Fix500ErrorsScripts/SI117426_list_all.sql
17) Start Vista
18) Checked 1071-5547 and it is fixed

January 31, 2007 10:25 AM - 11:05 AM
JMS failing to connect to prd2/prd3?
The prd2 and the prd3 vista servers were restarted. End-users should not have been affected by this restart.

January 18, 2007 2:00 - 6:00 AM
Regular Maintenance
-Shutdown all nodes
-Cleaned the logs
-Ran perl createWebLog.pl
-Ran Bb supplied database fix scripts (fixOrganizerLinks-61930, fixOrgPgLayout_4.0.X_60511, and fixUniqNameMissingDiff_4_0_x_Oracle.sql)
-Started all nodes

January 11, 2007 2:00 - 6:00 AM
Regular Maintenance
-Shutdown all nodes
-Cleaned the logs
-Ran perl createWebLog.pl
-Initiated getInfo.pl script on each node to collect diagnostic information for Bb support
-Started all nodes

January 9, 2007 1:15 - 1:45
Prd3 JMS connection failure
The prd3 vista server was restarted to restore its JMS connection. End-users should not have been affected by this restart.

January 2, 2007 7:00 - 12:10 AM
Intermittent Performance
-Restarted admin2 server
-Rolling restart of PRD1 and PRD2

December 29-30, 2006 5:50 PM - 3:00 AM
APS Campus Power failure
-Started all nodes and DB when power restored

December 21, 2006 2:00 - 6:15 AM
Regular Maintenance
-Shutdown all nodes
-Started all nodes

December 20, 2006 3:30PM
Prd3 was not communicating properly with the rest of the Vista cluster, it was shutdown to be restarting during the regular maintenance on 12/21. There should have been no Vista outage and there should have been little or no impact to users currently on the system.

December 14, 2006 2:00 - 6:00 AM
Regular Maintenance
-Shutdown all nodes
-Cleaned the logs
-Ran perl createWebLog.pl
-Started all nodes

November 29, 2006 11:00AM - 11:40 AM
- Restarted prd2 node of the Vista cluster to remedy intermittent problems with users on this node. There should have been little or no visible impact on the end users due to this restart. Vista remained up and operational during this time.

November 16, 2006 2:00 - 6:00 AM
Regular Maintenance
-Shutdown all nodes
-Installed Oracle Critical Patch Update for October 2006 (5490841) and Oracle patch 4199559 related to async I/O on Bryce and Chaco. This was done in preparation for the AP1 upgrade which will likely happen in the coming weeks.
-Cleaned the logs
-Ran perl createWebLog.pl
-Started all nodes

November 7, 2006 10:57AM - 11:16 AM
- Recycled Vista cluster due to poor performance and intermittent instability.

November 2, 2006 2:00 - 6:00 AM
Regular Maintenance
-Shutdown all nodes
-Updated browserchecker.xml file to allow Internet Explorer 7.0 and Firefox 2.0 to pass the browser check for the Windows operating system only.
-Cleaned the logs
-Ran perl createWebLog.pl
-Added the REFRESH file to $VISTA_HOME
-Started all nodes

October 19, 2006 2:00 - 6:00 AM
Regular Maintenance
-Shutdown all nodes
-Ran fix_dis_msg_delivery_content to fix some 500 errors
-Removed all '/' from both the override DB and the learning_context table
-Cleaned the logs
-Ran perl createWebLog.pl
-Added the REFRESH file to $VISTA_HOME
-Started all nodes

October 18, 2006 4:30PM
Restarted Chat on Prd3
-Killed the chat process on Prd3. There were some strange chat problems that were fix by this and all clients should have reconnected.

October 5, 2006 2:00 - 6:00 AM
Regular Maintenance
-Restarted all nodes
-Ran FixOrganizerLinks to clear up problems in some sections. Found no errors.
-Used new delay function for our standby for this outage for another level of disaster recovery
-Some of the name of our sections had an invalid character of '/' in the names. Replaced all '/' with '-'.
-Ran fix 500 errors script (SI117426_fix_all.sql) to clean up ACL issues.

September 28, 2006 2:00 - 6:00 AM
Regular Maintenance
-Restarted all nodes
-Ran FixOrganizerLinks to clear up problems in some sections
-Installed new GenericOutoingSSO.jar to allow for more robust error logging capabilities.

September 25, 2006 9:00 AM
Prd2 has begun producing errors, it will be removed from service so that Bb technical support can attempt to troubleshoot it while it is in an un-healthy state.

September 21, 2006 2:00 - 6:00 AM
Regular Maintenance
-Placed REFRESH file to refresh each node's configuration in an attempt to reduce the rate of server 500 errors.
-Restarted all nodes.

September 18, 2006 9:00 AM
Prd2 was returned to service and Prd3 was restarted in an attempt to reduce errors that were determined to be specific to Prd3.

September 17, 2006 4:15 PM
Prd2 was removed from service after determining errors related to this specific node.

September 14, 2006 2:00 AM - 6:15 AM
Vista down for scheduled power outage to continue work on UPS.
Change made to java initialization to give chat application more memory which should increase stability of chat service.

August 31, 2006 2:00 AM - 2:46 AM
New elluminate plugin was installed
Changed e-mail address in ./serverconfs/log4j.properties to use acad-ole@lists
Chat now working

August 30, 2006 3:00 PM
prd3, a new node, was added to the Vista production cluster. Service was not interrupted.

August 29, 2006 9:44 AM - 10:31 AM
Vista was unavailable during this time because of an operating system malfunction on the Vista stand-by database server, Chaco.

August 24, 2006 12:32 PM - 1:58 PM
Vista was down due to power failure on its disk array. APS power was lost due to lighting strikes. Vista is now back and running normally.

August 24, 2006 2:00 AM - 04:50 AM
Run DB Check
Reboot all nodes
Change admin email address from root@node to acad-ole@lists.nau.edu
Ran fix organizer links - Found no errors
Rotate and backup logs
Installed Elluminate PowerLink
Installed Horizon Wimba Voice Tools
Deployed new login page

August 3, 2006 2:00 AM - 3:05 AM
Applied Oracle Critical Patch Update CPUJul2006
Applied FixOrganizerLinks patch which repaired 117 links

July 24, 200611:00 PM
Changed Mail tool attachment size
You can now attach up to 10,000KB (About 10 megs) to a mail message

July 20, 2006 2:00 AM - 5:00 AM
Installed Service Pack 3
Update CAS Login URL
Re-enabled SSL on both managed nodes

June 29, 2006 2:00 AM - 2:45 AM
Installed Spanish Language Pack
webct1200_sp1_lp-es-108.zip

June 22, 2006 03:00
Bounce, TurnItIn, SSL Settings
- Shutdown Cluster
- "SSL Listen Port Enabled" set to false for all nodes
- Changed WLS_HOST in setEnv.sh from DNS name to ip in $VISTA_HOME/setEnv.sh
- Installed newest version of TurnItIn
- Restarted Cluster

June 20, 2006 01:30
Update JDBC Drivers to Fix Outages
Maintenance Completed 01:45
- Replaced $VISTA_HOME/serverlibs/oracle/ojdbc14.jar version 9.0.2.0.0 with version 10.2.0.1.0

June 19, 2006 4:10 PM - 5:27 PM
Vista down due to DB connectivity failure, restarted full cluster

June 19, 2006 7:35 AM - 1:50 PM
Vista down due to DB connectivity failure; switched to standby DB, rebooted the admin server as well as both Vista nodes, restarted full cluster. Opened Oracle TAR on protocol errer.

June 18, 2006 9:30 PM - 10:30 PM
Vista down due to DB connectivity failure, rebooted the admin server as well as both Vista nodes, restarted full cluster

June 16, 2006 1:30 PM - 4:45 PM
Vista down due to DB connectivity failure, rebooted the admin server as well as both Vista nodes, restarted full cluster

June 16, 2006 7:44 AM - 9:24 AM
Vista down due to DB connectivity failure, restarted full cluster

June 15, 2006 2:24 PM - 3:45 PM
Vista down due to DB connectivity failure, restarted full cluster

June 13, 2006 1:45 AM - 2:00 PM
Node B exhibiting writing problems due to DB connectivity failure, restarted full cluster

June 1, 2006 02:00
Install Service Pack 2 and Hot Fix 60901 and 61100
Maintenance Completed 05:28
- Broke Standby Mode
- Install SP2
- Updated weblogic.xml
- Archive Logs
- Update user_config.xml
- Start Cluster
- Install HF 60901
- Start Cluster
- Stop Cluster
- Install HF 61100
- Update vista.jks (Web SSL)
- Update vista.nau.edu.cert and vista.nau.edu.clear.key (Chat SSL)
- Move Time Zone, Generic Outgoing Deployable Component
- Start Cluster

April 6, 2006 02:00
Install Turn It In and Log Maintenance
- Installed Turn It In which required an application reboot
- While down archived logs

March 23, 2006 02:00
Down for EMC Maintance
- FLARE Operating Environment on the SAN (requires reboot of both controllers)
- Navisphere Agent software on both acadia and bryce
- PowerPath software on both acadia and bryce

March 2, 2006 02:00
Install 53490
Down to install Hotfix 53490. Back up by 03:00

February 23, 2006 02:00
Install new License
Down to install new License file. Back up by 02:15

February 16, 2006 02:00
Glossary Fix and New Spanish Language Pack
Down to run SQL fix to make broken glossary show back up and install new Spanish Language Pack. Back up by 02:15

February 02, 2006 02:00
Oracle 9.2.0.7 Patch Set
Down for Oracle 9.2.0.7 Patch Set. Backup by 03:00

January 26, 2006 02:30
Hotfix 60090 installed
Installed Hotfix 60090.

January 26, 2006 02:00
Oracle Critical Patch Update CPUJan2006
Our DBA's installed Oracle Critical Patch Update CPUJan2006.

January 19, 2006 11:00AM
Cluster Reboot
There was a problem when people would would upload a file the replace command would not work. The reboot was suggested by Support

January 10, 2006 11:00AM
Prd1 and Prd2 not syncing
It seems that an SSL issues was causing Prd1 and Prd2 to not talk togather. I change at the Weblogic level to not check host names fixed the issue. Since both Prd1 and Prd2 use the Vista.nau.edu SSL Cert, Prd2 did not accept the cert because it was coming from Prd1 not Vista.nau.edu. The change was made and Prd2 was entered back into the cluster.

January 5, 2006
Reboot to solve Java SSL issue
Updated the SSL Cert for Chat and White Board and that required a reboot.

December 23, 2005
Initial Server Installation
This server was installed and Upgraded to SP1 and Hotfix 53340.

  • No labels