Are Chase’s internal systems down?

Received this from a reader at 12:36 PM EST today:

For the last 40 minutes, each time I call Chase to take action on a pending transaction in my account, I’m told by the “telephone banker” that the systems are “updating” and they cannot access customer’s accounts.  Updating?  Really? On a weekday during business hours?  Their online site is functioning, but I find it odd that internal systems are not.

Is this another one of Chase’s unscheduled scheduled maintenance windows?  It would obviously be odd for any financial institution to be updating their systems on a planned basis in the middle of the day.

Mysterious Chase ATM outage

One customer’s deposits were never recorded because of what a Chase employee called an ATM outage.  Gee, is this kind of thing common with Chase?

My iPhone: DING! DING! DING!!!

Half awake, I pick up the phone to find out my account is several hundred dollars in the negative via text message!! Odd as I had just deposited several hundred dollars THE DAY BEFORE!!And what little funds remained in my account, where had those gone? Lost in space!! I go to being half awake to full blown panic as I wipe the sleep from my eyes!! How can this be!?!?!

My iPhone: DING! DING! DING!!

Another text message indicating that my account is even further into the red! What the hell ladies and gents, what the hell!?!?!

I call Chase to find out that they have no record of my deposit. I say I have a receipt and pictures on the receipt of my checks to prove it!!! The Chase representative sees something that are, in fact, my deposits!!! As I am on the phone trying to get this sorted, I see that Chase not only lost my money but had the gall to blast me with THREE INSUFFICIENT FUNDS CHARGES TOTALING MORE THAN 100 DOLLARS!!! It’s as if Chaseknocked me to the ground, pissed on me, then held me down for a few moments more to defecate on me!!!

I’m sent to the claims department and told that there was an ATM outage that caused anyone who deposited their money via ATM to lose it. They assured me over the next several days, they will reimburse me for everything including the insufficient funds fees.

The companies motto on ads that I see on TV are “Chase what matters.” Chase I did; MY MONEY LOST AT YOUR BANK!!!!

Is Chase down AGAIN?

We are receiving reports, as of 12:30 ET, that people are unable to log into their Chase accounts and one person sent us this link to an outage message (shown below).

Is Chase.com having problems again?

One reader wrote us this morning at 9 am ET:

I logged onto my credit card account this morning. It let me in but everything says “Not Available”. That is the balance, payment due, minimum due, etc.

Is Chase having troubles again?

Incidentally, the apologies on Chase’s front page from the last outage are gone now.

More details on Chase’s website crash emerge

From this article, much of this information seems to be from tips from a Chase insider.

There was a subsequent outage on Wednesday, apparently due to the huge number of access retries after the initial restoration of service.

There was a definite operator-error contributing cause, perhaps the error would not have happened if people weren’t so exhausted from dealing with the database outage.

Simply said, Chase’s website was not adequate to handle all the traffic when a larger than normal percentage of Chase customers tried to log on, and many people were unable to successfully log on for much of Wednesday, and that an error by Chase technical personnel exacerbated this problem.

Monash said JP Morgan Chase runs its user profile Oracle database on a cluster of eight Solaris T4520 servers, each with 64GB of RAM, with the data held on EMC storage. El Reg is told that Oracle support staff pointed the finger of blame at an EMC SAN controller but that was given the all-clear on Monday night.

Monash subsequently posted that the outage was caused by corruption in an Oracle database which stored user profiles. Four files in the database were awry and this corruption was replicated in the hot backup.

Recovery was accomplished by restoring the database from a Saturday night backup, and then by reapplying 874,000 transactions during the Tuesday.

For the non-technical folks in the audience, a piece of storage hardware failed and subsequently caused the databases to get corrupted in both the live and real-time backup.  Most databases of this type have many layers of backup, and that was the case here.  In addition to periodic backups, a typical system will keep a “journal” of any activity that is applied so that in a worst case scenario like this, the list of database changes can be applied and the data from the older backup can be updated with all subsequent changes.  But it can take some time.

It seems likely that at some point during the outage Chase must have known what was going on and that they would eventually be able to fully restore the service and no data would be lost, which makes it even more perplexing that they didn’t release any statements to this effect.

This Oracle database stored user profiles, which are more than just authentication data.Applications that went down include but may not be limited to:

• The main JPMorgan Chase portal.
• JPMorgan Chase’s ability to use the ACH (Automated Clearing House).
• Loan applications.
• Private client trading portfolio access.

So, clearly more than just account access and bill-pay were affected.  ACH transactions include things like paying an Ebay auction using PayPal, which comes from your checking account.  They should have released information better telling people what was and was not affected while the outage was occurring.

Chase not being flexible with customers hurt by the outage

On Wednesday, a Chase official made the following statement:

“If a customer incurred a late fee during this outage, they should call us via telephone banking or go into a branch, and we will rescind that late fee,” she said. “If they incurred these fees during the outage because they could not make a payment any other way, then it is our responsibility and we will refund that late fee.”

The language sounds like it is specifically crafted to give them an easy out by just claiming a customer didn’t try every other way to make a payment, and it looks like they are already availing themselves of this.  Here is a comment we received:

I finally was able to get in yesterday afternoon as I needed to pay my 2 cards – they were due on the 15th.  It was after the 5 p.m. EST cut off time (I’m on PST).  I called and got customer service, as I wanted to know if I could pay at a branch. My only option: pay $14.95 to have them take the payment and process effective the due date. $14.95 per credit card. I wanted to know how they could charge this fee when they have been down. Her response: you had the option to call our number and pay via our automated system before 5 p.m. EST.  Great.  I always pay online from where I’m at. I didn’t have any of my info (phone #, credit card #, bank account #).  Seriously sucks.

C’mon Chase, you seriously messed up and not everyone had time to wait on an extended hold on you seriously backed-up telephone banking or make it into a branch.  You should be waiving fees like this left and right.

Chase online failure – after effects

The comments and Twitter are quiet about new problems so I think we can assume that the full (Monday, Tuesday) and partial (Wednesday) outages of Chase’s online banking are over for now.

The Wall Street Journal has a good article on the problems worth reading.

Chase’s biggest failure throughout this mess, and in its aftermath, is the failure to inform its customers what is going on.  First of all, they tried to pass it off as scheduled maintenance, in the middle of the day.  Then they didn’t officially acknowledge the problem, other than the notice on their site.  When problems reappeared on Wednesday, they were very slow to acknowledge them.

During the outage the very limited information they released did not indicate anything about the nature of the problem, how people might be ultimately affected, and how long the problem might last.  They failed to use the best possible channel to get any information to their customers, their website, to release information.

Lastly, since the problem was fixed, they have not come out with a clear explanation of the cause of the problem and what, if any, customer data or worse, funds, might be at risk, or have indicated that such a statement will be forthcoming.  The only information they provided to customers after the fact was a short apology and an incomplete description of what would happen to one type of transaction scheduled during the outage.

In short, this outage has been a complete communication failure for Chase as they left their customers completely in the dark.

We are getting a limited number of reports from people who are finding fraudulent activity in their accounts.  It is unclear at this time whether this might be a more widespread problem, whether this is normal fraud that is incorrectly being associated with the outage, or whether what appears to be fraudulent activity was caused by data corruption.

A short while ago they updated the message available from their main page from the previous short apology to a longer message similar to that which was available to customers once they logged in.  Here is the message:

We are sorry for the difficulties that recently affected Chase.com, and we apologize for not communicating better with you during this issue. Giving you 24-hour access to your banking is of the utmost importance to us. This was not the level of service we know you expect, and we will work hard to serve you better in the future and to communicate with you better if a situation like this should arise again.

Online Bill Payments scheduled for September 13, 14 or 15 were processed by Wednesday night, September 15. It is not necessary to reschedule these payments. If you scheduled a payment during those dates, but do not see it reflected in your payment activity by September 16, please contact us.

We will refund any late fees that you may have incurred as a result of our delay in processing your payment.

Thank you for your patience and for the opportunity to work harder to serve you in the future.

They are now additionally apologizing for the lack of communication and claiming to do better in the future.

This site (chase-sucks.org) was created in the hopes of helping convince Chase that they needed to do better at serving customers (more service, less shenanigans).  I can only hope that the black-eye this outage, and their response to it, has given Chase will truly convince them to do better for their customers.

Update Thursday 9/16/10:  Chase has not change the top banner on their main page to duplicate the apology link, as if saying it twice makes it carry more weight.  The apology text itself has not changed.

Update Saturday 9/18/10:  Here is a report from a customer saying their debit card stopped working as a results of the Chase outage:

I tried to use my debit card for an online transaction today and it was declined.  This immediately worried me given the problems they’ve been having, so I tried to withdraw money at an ATM.  An error message came up at the ATM saying that my card information was “incomplete.”  Whatever happened to their system has clearly scrambled their databases.  I live overseas and rely on my ATM card to access my money.  No working debit card=no way for me to get money, except for an expensive credit card advance.  The best part?  They are saying that they need four or five days to even get back to me on this.

Chase back up for good?

Sometime around 9 or 10 pm ET or so Chase online banking seems have stabilized, which we are estimating by when we got our first report in the comments and when the “Chase is down” tweets stopped.  Chase hasn’t provided an official time when the problem was fixed.

They have posted a short apology on their website but it contains no info on the cause of the problem:

To our customers:

Please accept our apology for the difficulties that recently affected Chase.com.

Giving you 24-hour access to your banking is of the utmost importance to us. This was not the level of service we know you expect.

Thank you for your patience and for the opportunity to work harder to serve you in the future.

Their spokesman released a brief statement about the nature of the problem:

“A third-party database company’s software caused a corruption of systems information, disabling our ability to process customer log-ins to chase.com. This resulted in a long recovery process,” Chase spokesman Thomas Kelly said Wednesday evening.

Clearly log ins were affected.  It remains to be seen whether reports of other fallout problems will surface when people start using the service in earnest on Thursday.  A more detailed statement was apparently available to customers once they logged in:

Please accept our apology for the difficulties that recently affected Chase.com.

Online Bill Payments scheduled for September 13, 14 or 15 have been processed or will be processed Wednesday night. It is not necessary to reschedule these payments. If you scheduled a payment during those dates, but do not see it reflected in your payment activity by September 16 please, contact us.

We will refund any late fees that you may have incurred as a result of our delay in processing your payment.

Giving you 24-hour access to your banking is of the utmost importance to us. This was not the level of service we know you expect.

Thank you for your patience and for the opportunity to work harder to serve you in the future.

It is also highly likely that the Chase’s online banking system will be getting higher than usual volume Thursday as people log in to check their accounts.  It will be interesting to see what happens.