Panopto EU Cloud full outage

Incident Update

Panopto Cloud Operations Status

Current Status: Operational

Affected Infrastructure
Components: Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations
Locations: Europe (EU)

On 2021-08-11 from 21:48 UTC to 22:07 UTC, the Panopto EU Cloud had a full site outage. All users were unable to access their sites, including being unable to view, edit or upload new recordings.

RCA: Starting at 21:48 UTC, the Panopto EU Cloud web servers received a surge of incoming requests. Each of these requests required handling by a subservice, and thus resulted in requests being sent from the web servers to that subservice. Due to both the large quantity of requests and the processing cost of handling these requests, the subservice quickly became backlogged leading to elevated latencies in handling requests. Complicating that, a configuration flaw in the web servers made it so these long-running requests were not terminated in a timely manner, and the web servers quickly became backlogged and unable to handle general site traffic. Our operations team mitigated the issue by manually scaling servers to exceed the demands of the temporary surge of requests.

To prevent a recurrence of this issue, our cloud operations team will update the web server configurations to ensure subservice requests have appropriate timeouts.

Panopto New Features and Important Summer Dates

Panopto Logo

 

 

Panopto Downtime:

On Saturday, June 26th Panopto will be down for maintenance and for the installation of new features for the service. They believe the downtime will only run for about 4 hours, and will start at 10pm BST on the 26th.

During the downtime, you will not be able to access videos on your Panopto site and any attempts to upload from recording clients will result in a “Server unable to connect” message

Although it’s unfortunate that the service will be down, the features that will be arriving upon the upgrade are quite nice, and many have been things that staff have been requesting/hoping for for a while.

New Features:

Multi-stream mobile and embed viewer — Exciting changes are coming to mobile devices and the Panopto embed viewer. Users will get a fully interactive multi-stream experience that allows them to dynamically control which streams are displayed and to cycle through visual layouts. Linked Youtube videos are also now supported in this viewer, along with the full suite of features available in Panopto’s interactive desktop viewer.

Discussion Moderation — Content creators can moderate discussions for both live webcasts and on-demand videos, controlling which discussion comments are visible to the full audience of viewers.

Multi-stream preview in Capture — After recording a video with Capture, users will now see all of their recorded streams in the preview viewer to increase confidence that all of their content has been captured correctly.

Improved Sharing Experience — In response to usability feedback from users, we have revamped the sharing experience to be more intuitive and to use a modernized look and feel.

Multi-Language Search — Users can specify the content language for each folder, enabling Panopto’s Smart Search and speech-to-text processing in multiple languages within the same site.

Join or resume a session using Capture — Users can now use Capture to create distributed recordings, joining an in-progress recording or resuming a previous recording, right from the video library.

Improved clips ordering — Users can now drag and drop clips to the desired location from the timeline UI in the editor.

Mobile apps: Background/Reliable Upload & Share — Panopto’s mobile app now allows users to create recordings of unlimited duration and will upload videos while the app is in the background. Users can also easily share videos from within the app.

Some of these features will take a bit of time to wrap our heads around them, but Panopto believes that their service will be even easier and better to use.

Mandatory Client Upgrade

On Friday, July 16, we will require Panopto for Windows & Remote Recorders to be upgraded to version 9.0.2 or above and Panopto for Mac to version 10.1.0 or above.
The upgrade will be applied to your Panopto site at 12 midnight GMT/1:00 am BST.

To update your Panopto Recorder, go into a module with Panopto Video enabled. Click on Panopto Video to go to the Panopto Content area of the module (1) and then click on the Go to Panopto button in the upper right (2).

Screencapture of a module area with Panopto Video link and the expand button being highlighted

Once you’re in the Panopto Area, click on the “Download” button under your name in the top-right.

Screencapture of the download panopto video button under the users' name

Click the green Download button.

Screencapture of the green coloured download button

Double-click to launch the installer application

Screencapture of the downloaded exe file installer

Follow the onscreen-prompts to install the software.

Note: If you are prompted to enter a password to install this update, you MAY have to contact the Service Desk if the computer doesn’t allow you to run the installer.

RESOLVED: Panopto Recorder version 9 for Windows – Known Issue

Panopto have released an update to the Recorder for Windows that fixes the issues.

Last week we discussed the bug affecting Panopto for Windows and PPT. I wanted to let you know in case you missed the updated in the forums, this was fixed in Friday’s release:

We have had reports from academic staff about issues with Panopto Recorder when recording Powerpoint lectures, where the slides are NOT recorded and the resulting video is simply a blank screen with the lecturer’s voice only as a ‘voice over’.

In diagnosing this issue, we have discovered that it is a known-bug (problem) that is being worked on by Panopto Engineers.

Our advice:

  1. Do not update to Panopto Recorder for Windows 9. No matter how often you are prompted to do so. Just ignore the prompt.
  2. If you’ve already updated, Choose to record BOTH from Main Screen and Powerpoint
    Screencapture showing both the Main Screen and Powerpoint tickboxes ticked in the Secondary Source area of the window
  3. Or, to download the Powerpoint from your OneDrive onto the desktop of your local computer and run the presentation from your computer. The issue only effects Powerpoint files that are located in a networked drive, like OneDrive or DropBox.

Here’s the link to the Panopto Community site discussing the issue:

Community Panopto

Issues affecting Panopto Video search capabilities Jan-Feb 2021

Panopto is cognisant of continued issues with their ‘Search Functionality’ across the EU and Britain.  They have engineers building new systems, and server cascade responses to mitigate the issue.

Every three or so days since January 20th, we’ve been receiving these messages from Panopoto:

Incident Update
Panopto Cloud Operations Status

Current Status: Partial Service Disruption

Affected Infrastructure
Components: Search
Locations: Europe (EU)

Update:
The Panopto EU Cloud is experiencing an increased error rate in search.

The issue seems to be ongoing. If you have issues ‘searching’ your videos, please be patient, the ability will most probably return upon the next time you log in to My Studies.

Panopto Service Disruption 22nd Dec, 2020

On Dec 22nd 2020 between 22:05 and 22:25PM GMT, the Panopto EU Cloud experienced a site outage.

Panopto’s cloud operations team has resolved the issue.

The cause of the incident was a misconfiguration on a database server. During Panopto’s most recent downtime release on Dec 19, they updated the configuration of their database servers to better utilize physical memory. The new configuration contained an error that caused the database server to reach an out of memory condition. Panopto resolved the outage by fixing the misconfiguration on the server that ran out of memory. To prevent a recurrence of this issue, their engineering team audited and updated the memory configuration on all database servers.

Panopto Service Disruption 20-10-20

Incident Status

Service Disruption

Components

Website, Video playback, Search, Recording / Uploading, Processing / Encoding, Integrations

Locations

Europe (EU)


October 20, 2020 11:41AM UTC
[Investigating] We are investigating increased error rates on requests to the Panopto EU Cloud.
October 20, 2020 12:15PM UTC
[Investigating] We are continuing to investigate the cause of increased error rates on requests on the EU cloud.
October 20, 2020 12:31PM UTC
[Monitoring] The source of the increased error rates has been mitigated. Our cloud operations team is monitoring performance metrics to ensure the issue is resolved.