September 2016

Observed Slowness of Visitor Status Changes
Visitor status should be operating as normal. Thanks for your patience!
Sep 19, 12:27-14:35 EDT
Cannot add new Operators
We've identified and resolved the operator issue. Thanks for your patience!
Sep 15, 17:23-18:03 EDT
Chatboxes offline
We have observed that the issue is no longer occurring since we restarted the servers. We got the 'all clear' at this point.
Sep 15, 09:09-10:35 EDT
[Scheduled] Server Maintenance Tonight
The scheduled maintenance has been completed.
Sep 7, 22:00-22:41 EDT

August 2016

Chat console disconnect warnings
Our chat console is now operating normally.
Aug 10, 09:03-10:27 EDT
Email, transcripts and integration issues.
All of our transcript and email systems are back to normal and operational.
Aug 4, 15:22-16:54 EDT

July 2016

No incidents reported for this month.

June 2016

Degraded Performance - Cobrowsing
We've just got updated that the issue with Cobrowsing should now be stable.
Jun 23, 16:13 - Jun 24, 14:19 EDT
Cobrowsing unavailable
Cobrowsing is back up and running. Thanks for the patience.
Jun 23, 13:29-14:12 EDT
Network outage
We're no longer seeing reports of Olark not loading on sites affected by the European ISP outage earlier today. We're continuing to monitor the situation but it we are resolving this incident for now.
Jun 20, 08:45-13:53 EDT
[Scheduled] Server Maintenance Tonight
The scheduled maintenance has been completed.
Jun 15, 12:48 EDT
Chatbox not loading on sites
We're confident that the issue has been resolved. We've seen no further occurrences in the last 30 minutes, therefore marking this incident as resolved.
Jun 3, 09:39-11:01 EDT

May 2016

Backing service outage
Our services are stable and fully operational. Thanks for the patience.
May 26, 23:09 - May 27, 02:42 EDT
Partial outage of Visitor Details {{geolocation}}
We are confident the issue with Geolocation is now resolved. The change is propagating and you will see geolocation information shortly if you are not already. Please do tell us if you continue to have issues.
May 26, 04:21-05:52 EDT
Legacy Chatbox Partial Outage
A chatbox code change we deployed at 1:08pm Pacific caused a host resolution problem on one of our legacy loaders which currently a very small subset of clients are using. During this time it’s likely the chatbox would not have loaded properly. We rolled back this change at 1:56pm Pacific and consider this issue resolved.
May 25, 17:24 EDT
Chatbox conflict with underscore library
For a period of about one hour starting around 12:20pm PST, the deployed chatbox Javascript code contained an external dependency overriding the "_" variable in the global namespace, affecting some websites using the underscore library. We rolled back the changes immediately, and as of the rollback all new downloads are no longer affected by this issue. Previously loaded chatboxes may have contained a cached copy for up to 2 hours. Although we have safeguards in place to prevent this, today's issue was caused by a gap in automated testing of final builds. Future builds will now run an additional simulated runtime check for all globals before allowing a deploy.
May 16, 21:53 EDT
[Scheduled] Olark chat console maintenance
The scheduled maintenance has been completed.
May 7, 22:31 - May 8, 00:30 EDT
Chat connection issue
We've been continuing to monitor the issue and are satisfied that the chat service has returned to normal. We're resolving this incident and operators can chat again as normal. Currently we believe the issue to have been caused by affected load balancers.
May 6, 11:38-13:51 EDT
Potential disruption of Olark service due to cloud provider maintenance
This incident has been resolved.
May 5, 01:09-08:57 EDT

April 2016

Transcripts issues
We believe that the issue with transcripts has been resolved and is now working normally.
Apr 12, 09:46-17:50 EDT
[Scheduled] Server maintenance tonight, April 6th, from 11:00PM EST - 11:30PM EST.
The scheduled maintenance has been completed.
Apr 6, 23:00 - Apr 7, 00:00 EDT
Chat Console Connectivity Issues
We've monitored services for a couple hours and did not see any further instability. At this point, we believe the issue is resolved. We will be providing a post-mortem update regarding the issue.
Apr 4, 17:47-22:26 EDT
Operator disconnection
All systems appear to be functioning as normal again. Operators can continue to log in and chat as usual. Similar to the incidents on March 30th (see: http://status.olark.com/incidents/zcsc15y53xwq and http://status.olark.com/incidents/2c36gzlv1qw7), we believe this to be caused by a service disruption to an upstream provider. Once the causes are clearer, we will write an update explaining what occurred as well as any steps we are able to take to reduce the risk of this happening again.
Apr 1, 14:36-15:51 EDT

March 2016

XMPP connection issues
We've had no further reports of XMPP chat connection issues, and our metrics indicate the chat is performing as normal. We're resolving this issue as we continue to investigate today's problems/
Mar 30, 14:56-15:32 EDT
Chat console connection
The issue with the Olark website and operator console has now been resolved. You should be able to use the service as usual. We believe this was caused by a service interruption for an upstream network provider, we're continuing to investigate. Thanks again for your patience while we addressed the issue.
Mar 30, 12:31-13:11 EDT
[Scheduled] Server maintenance tonight, March 24, from 11:00PM - 5:00AM EST.
The scheduled maintenance has been completed.
Mar 24, 23:00 - Mar 25, 05:00 EDT
Chatbox settings are slow to change.
Chatbox settings is now fully operational.
Mar 1, 07:45-08:43 EDT

February 2016

We're experiencing extra latency on chat network account creation and updates.
Chat network account creation and updates are now fully operational.
Feb 28, 17:04-18:15 EDT
Possible Message Delivery Delays
We're satisfied the issue related to message delivery has been resolved. We're continuing to look at the underlying issue that caused this to happen, but customers should not be currently affected.
Feb 18, 10:07-11:50 EDT
Cobrowsing issues
Cobrowsing functionality should now be resolved for all customers. We will continue to monitor this issue, but we believe it to be resolved now. Thanks for your patience! =)
Feb 16, 09:58-12:48 EDT
Transcript Service Interruption
Everything should be copacetic with transcripts!
Feb 11, 02:35-03:40 EDT

January 2016

Chat Service Interruption
We're resolving this issue as our metrics continue to look normal and we aren't seeing any further reports of problems. We apologize for this interruption of service.
Jan 27, 18:08-18:33 EDT
Internal networking issues affecting chat
We're resolving this issue as our metrics continue to look normal and we aren't seeing any further reports of problems. We'll be providing a postmortem of the events that led to the outage as well as how we might avoid this in the future. From a support point of view, we'd want to apologize for any delay in our response to emails or tweets. Our helpdesk provider was experiencing issues at the same time, making it difficult to reply to tickets in a timely manner.
Jan 26, 00:34-13:55 EDT
Chat.olark.com login issues
We are confident the issue has been resolved and are reflecting on the outage so we can take steps to prevent similar problems occurring in the future.
Jan 25, 10:41-14:05 EDT
Degraded performance in visitor details
This incident has been resolved.
Jan 19, 13:29-14:42 EDT

December 2015

No incidents reported for this month.

November 2015

Database issues
Around 2 PM EST one of our database servers had an issue, causing problems in several parts of Olark. Chat was affected for about 5 minutes. Olark.com was down for about 15 minutes, and chat.olark.com was down for about 25 minutes. All systems are currently operational again, but if you have questions please contact us at [email protected]!
Nov 12, 14:49 EDT
[Scheduled] Server maintenance tonight, November 3, from 11:00PM - 11:59PM PST.
The scheduled maintenance has been completed.
Nov 4, 00:53-00:54 EDT
Issue connecting to chat.olark.com
This incident has been resolved.
Nov 3, 11:09-12:20 EDT

October 2015

Server maintenance tonight, October 29, from 12:00AM - to October 29, 2:00AM PST.
Maintenance is over, all clear :)
Oct 28, 13:33 - Oct 29, 05:37 EDT
Issues with chat box loading and message delivery
This incident has been resolved.
Oct 28, 13:11-14:01 EDT
Server maintenance tonight, October 26, from 9:00PM - to October 27, 4:00AM CDT.
The server updates are now complete, have a good one!
Oct 26, 12:54 - Oct 27, 09:14 EDT
Issues with CRM integrations
This incident has been resolved.
Oct 23, 08:47-13:01 EDT
Server maintenance tonight, October 23, from 12:00AM - 2:00AM CDT.
No issues to report from server maintenance.
Oct 22, 17:32 - Oct 23, 08:39 EDT
Potential Interruption to Olark Services
Maintenance has been completed.
Oct 19, 15:25 - Oct 20, 11:47 EDT
Message delivery incident
The issue affecting a small number of Olark accounts and message delivery should now be resolved.
Oct 19, 14:52-15:29 EDT
chat.olark.com issues login in
We believe this issue to be resolved, and are continuing to monitor it.
Oct 13, 11:11-11:37 EDT
Olark operator console service degradation
If you are having trouble logging in to the operator console try restarting your browser to clear your DNS cache.
Oct 6, 11:50-13:03 EDT

September 2015

Site changes not being updated
Site setting updates issue has been resolved.
Sep 12, 11:54-14:32 EDT
chat.olark.com issues login in and with shortcuts
Systems are now stable, incident has been resolved
Sep 8, 15:23-19:22 EDT
[Scheduled] Backend infrastructure update
The scheduled maintenance has been completed.
Sep 3, 23:00-23:15 EDT

August 2015

Chat Console performance
Just before 2pm Pacific (5pm EST) one of our load balancer systems started experiencing issues. By 2:45pm Pacific we had spun up and switched over to another load balancer system and chat became operational again. The cause of the issue was a temporarily disabled load balancer, as a result of a DDOS attack on a site unrelated to Olark. As a result of the outage, our new load balancer now supports IPv6 and we've taken steps to add additional reliability around this part of our system.
Aug 25, 17:17-18:13 EDT
Login Issue
This incident has been resolved.
Aug 21, 05:34-06:53 EDT
Routine maintenance on chat.olark.com
Maintenance has been completed on chat.olark.com
Aug 13, 21:12-21:33 EDT
Desk.com service interruption
Desk.com interruptions have been normal for the last three hours or so. We are continuing to monitor the situation, but integration activity should be normal from that time period onward.
Aug 11, 15:57-20:04 EDT
Issues with Messages Being Delayed
From about 20:25 UTC to 20:32 UTC we were seeing slow message delivery, but the issue is now resolved.
Aug 10, 16:37 EDT

July 2015

Windows chat console connection issue
We've resolved the issue preventing operators connecting to the Olark chat console at https://chat.olark.com. We believe this is caused by a recent change in the Avast! Webshield as part of their antivirus product. We have found that disabling the "HTTPS scanning" feature within Webshield allows operators to connect to Olark. You can find steps on how to disable HTTPS scanning at: https://www.avast.com/en-us/faq.php?article=AVKB189 Olark is in communication with the folks at Avast! to address the problem permanently. If you're not comfortable disabling the HTTPS scanning, you can use Pidgin on your Windows desktop in the meantime. Find setup instructions for Pidgin at http://www.olark.com/help/pidgin For users seeing the "Whoops! We were unable to connect to the server..." error who do not use Avast!, please submit a support ticket at https://www.olark.com/help/ticket
Jul 22, 12:02 - Jul 27, 13:50 EDT

June 2015

[Scheduled] Olark chat console maintenance
The scheduled maintenance for chat.olark.com has been completed.
Jun 11, 22:00-22:53 EDT
Chat console connection issues
We believe the incident has been resolved. chat.olark.com should now be working as normal. We will continue to monitor the situation.
Jun 3, 04:52-05:02 EDT

May 2015

[Scheduled] Olark chat console maintenance
The scheduled maintenance has been completed.
May 14, 23:19 EDT

April 2015

Issues with Logging in to Olark with your Google Account
Earlier today there were some issues with users logging in to Olark using their Google account. Logging in to Olark with your Google account is currently working and we're continuing to investigate. If you see any odd behavior please contact us!
Apr 16, 14:52 EDT
[Scheduled] Website Login Maintenance
The scheduled maintenance has been completed.
Apr 9, 02:00-02:15 EDT
Transcripts incident
The issue affecting transcripts has been resolved. Transcripts should be appearing on the Olark dashboard and in your CRM integrations as normal from now. Thanks again for your patience.
Apr 7, 17:04-17:42 EDT

March 2015

[Scheduled] Backend infrastructure update
Scheduled maintenance complete. You can log in and chat as normal. Thanks for your patience.
Mar 31, 21:00-21:34 EDT
Connecting to chat.olark.com
All chat.olark.com servers are operational again. Thanks for your patience.
Mar 25, 10:49-11:15 EDT

February 2015

No incidents reported for this month.

January 2015

No incidents reported for this month.