Topic: Big data loss issue - PLEASE HELP

We are losing 10% - 25% of our incoming calls.  They show up on the Quickfuse billing statement (so we got billed for them) but don't show up (intermittently) in the Quickfuse database, our SOAP call, or both.   I have gone back through Monday of this week and have a spreadsheet with the 280 or so calls we were billed for and which ones made it to the various databases.  Can someone please contact us directly so that we can send that spreadsheet to help you figure out what the issue is?

Re: Big data loss issue - PLEASE HELP

Hi,

For these calls that you don't see data for, it may be because the caller has yet reached the portion of your application where it sends data to your data table.

We have created a clone of your application and will be placing test calls to see if we can reproduce the issue of data becoming lost.

Regards,
The QuickFuse Team

Re: Big data loss issue - PLEASE HELP

Thanks for replying.  I would say that might be a likely explanation, EXCEPT that for most of these calls we have a SOAP entry in our database.  The SOAP entry happens in post-call processing.  For almost all of these calls, we have all of the data entered, including a voice recording and a menuchoice.  The menuchoice module feeds right into the database entry module so if they entered a menuchioce then it should have saved immediately to the database.

Examples of where this happened are the following calls (from the QF billing output).  For each and every one of these we have a soap call that was made, and a menu choice, ID# and voice recording was made, but there is no entry in the Quickfuse database.
Caller ID                     Timestamp
[removed for privacy issues]

Last edited by cjenkinscpa@●○○○● (2011-06-17 18:25:22)

Re: Big data loss issue - PLEASE HELP

In addition, we have been getting a lot of people telling us that the timecard app is hanging or that the voice recording is just repeating and not moving forward - to us there is clearly an issue with some module or the servers.  We haven't changed our app at all.

Re: Big data loss issue - PLEASE HELP

Hi,

Thanks for this information.

We will attempt to reproduce these symptoms you have mentioned as we test your application.

Regards,
The QuickFuse Team

Re: Big data loss issue - PLEASE HELP

Hi,

We have looked into this issue and have found the following:

1) About the timecard application hanging and the voice recording repeating not moving forward, we have placed numerous calls into the application and have been unable to reproduce this issue. Has this issue occurred for you while testing?

2) We have been able to reproduce this data table issue you've experienced. Our engineering team has looked into it and have noted that you are using the Simple Database built into QuickFuse.  This Simple Database system does not come with any data storage guarantees.  We strongly recommend that you store any critical data using a web service with the SOAP or REST modules and only use the Simple Database tables within QuickFuse for prototype applications.

Sorry for this inconvenience.

Regards,
The QuickFuse Team

Re: Big data loss issue - PLEASE HELP

Our customers using the application (actually calling in and logging time) are the ones that are experiencing the issues.  We have had many calls and I have been able to re-create it once by calling in (not testing and having the system call me) although it is clearly an intermittent issue.  Comparing the Billing log caller ID to the SOAP database calls over the last 24 hours, we are losing about 25% of the data (i.e. a call is made but the SOAP call never gets made).  As the post-call handling should always include a SOAP call, this is clearly an issue.

In addition there is about 50% loss on the Quickfuse database side.  In the past the Quickfuse database has been very reliable (until sometime this week).  I'm not looking for any data storage guarantees - I'm asking why the database module is not functioning properly in the application.  It is simply not triggering a database write event in almost half of the cases.  Why is this?  We use the Quickfuse database to provide a confirmation and back up to the SOAP database but since it's not working correctly we are clearly losing data.

Please help us figure this out - it is very likely an issue with your application/modules and it's not much help to say "we don't offer data storage guarantees".  Please help!

Re: Big data loss issue - PLEASE HELP

Hi,

Our engineering team is still looking into whether there are any issues with the data table modules. However, in the meantime, we recommend altering your application to use the SOAP webservice module to store your data as the data tables are unreliable for storage (and should only be used for prototype applications) and do not guarantee that data will be stored within them.

Regards,
The QuickFuse Team

Re: Big data loss issue - PLEASE HELP

Thank you for responding.  We are still having big issues with data.  We have always used the SOAP service, with the QuickFuse simple database for backup and checking whether we received all calls through the SOAP interface.

We and our customers are still having issues with the system.

Here is what we have done since Friday:
1- Removed the simple database calls from the application entirely (the database was completely unreliable)
2- Cloned the application and re-recorded all prompts (this let to the error messages reported in another thread.  It took 5-6 tried to record each prompt as your system clearly had trouble saving the prompts once recorded).
3- Validated whether SOAP was working.  It currently does not work correctly in 10% of cases.  This is still happening.
4- Made several test calls via the "Run" function of the application editor as well as calling in to the timecard system.  Several of these calls yielded long pauses or error messages.  Some of these did not result in proper post call handling.

In summary, we are still losing calls and data.  I have experienced, and customers have reported, delays in getting to subsequent prompts, error messages, and incorrect/incomplete post-call handling (no SOAP).  In comparing our billing record (what your Account screen shows) and our SOAP database it appears we are losing about 10% of calls right now.

Re: Big data loss issue - PLEASE HELP

Since Monday morning we have lost 2/7ths or 29% of our calls based on your billing records.  No SOAP call was made in these cases.  Our customers are calling us asking why the application is freezing and they do not think their time was recorded properly (it wasn't).  Please review your servers to see why this is happening!!!!

Re: Big data loss issue - PLEASE HELP

Hi,

Our engineering team is continuing to look into what is causing these issues when using any of the database modules.

In the meantime, we suggest using a SOAP webservice module within your application to replace your Insert Into Database module and sending your data through this method. We have tested a cloned version of your application that had replaced your Insert Into Database module with a SOAP webservice module and these have not produced any data issues.

Sorry again for this inconvenience.

Regards,
The QuickFuse Team

Re: Big data loss issue - PLEASE HELP

Thank you for your suggestion.  We have already done this but are still experiencing issues.  We have 2 different SOAP approaches implemented currently. We have put a SOAP module in the application.  We also have a post-call processing SOAP database service. 

Here are the intermittent issues we are still experiencing today:
- Customers are still complaining.  They are having long pauses between prompts or perceive the application as "hanging."  However in some of these cases, we end up getting the SOAP call to our database and get the data, but then the customer feels they have to call back in because they think the system didn't get their call. 
- Sometimes the system does not hear the customer's voice in the voice recording - it perceives it as silence, although the customer is clearly speaking into the phone and there is no mute.
- We are still missing some SOAP calls.  Sometimes the SOAP module in the application is successful but not the post-call processing SOAP service, and sometimes the other way around.  Sometimes they are both successful.   In a couple of cases, neither is successful.

I appreciate that you are working on this - is there any way that someone from your team can call me to discuss this ongoing issue?

Re: Big data loss issue - PLEASE HELP

Do you guys have any update?  We are still experiencing the same problems.  I have tons of data to share about which calls are being lost if someone would like to contact me directly.

Re: Big data loss issue - PLEASE HELP

Hi cjenkinscpa,

Our engineering team is still working on the issues occurring for the data table modules. We will provide you with an update on that situation as soon as we hear from them.

Also, we haven't been able to reproduce the data loss issues that you have seen while using a cloned version of your application using the SOAP webservice module with our own webservice.

However, within your application, we suggest attaching the "Service unavailable" and "SOAP Fault" nodes of your SOAP Webservice module to a different module (instead of the Branch on Number module) as there would be no way to tell if there was an issue with your webservice in attempting to send the data.

Regards,
The QuickFuse Team

Re: Big data loss issue - PLEASE HELP

admin wrote:

However, within your application, we suggest attaching the "Service unavailable" and "SOAP Fault" nodes of your SOAP Webservice module to a different module (instead of the Branch on Number module) as there would be no way to tell if there was an issue with your webservice in attempting to send the data.

I'm happy to try that.  Can you please tell me what module or setup you would suggest attaching those nodes to in order to help in debugging?

Re: Big data loss issue - PLEASE HELP

Normally I would make a database call to store an error but your database doesn't work and I'd have to do another SOAP call.  Since we're trying to debug a SOAP failure that doesn't seem to make sense.

Re: Big data loss issue - PLEASE HELP

Hi,

Regarding your question, you should add a Simple Prompt module to the "Service unavailable" node of your module telling the user, "The web service is currently unavailable. Please try your call again later." For the "SOAP fault" node, you could similarly add a Simple Prompt module that tells the user, "The SOAP web service has experienced a SOAP fault. Please try your call again later."

Regards,
The QuickFuse Team

Re: Big data loss issue - PLEASE HELP

That is not a practical approach for us and does not help us in debugging.  That's what I am looking for here.  Our customers using a timecard system do not know what a SOAP fault is!

Have you found out what your problem is with these modules yet?  We are now at 2 weeks of problems and the data loss is still occurring.

PLEASE HELP!!!

Re: Big data loss issue - PLEASE HELP

By the way - not only is the data loss (SOAP problems) still occurring but our customers are having various pauses, etc so they end up calling in a second time, even if the call did actually go through.  Something is clearly not going correctly on your servers.

Re: Big data loss issue - PLEASE HELP

Hi,

Our engineering team is still looking into your issue. Once we have an update from them, we'll be sure to let you know.

Regards,
The QuickFuse Team

Re: Big data loss issue - PLEASE HELP

Any update?  We are still losing about 10% of call data, despite having SOAP calls both inside the application and as part of post-call processing.

Re: Big data loss issue - PLEASE HELP

Hi,

Our team is still working to resolve this issue. We will let you know as soon as we have a resolution.

Regards,
The QuickFuse Team

Re: Big data loss issue - PLEASE HELP

Since the Total system outage on 7/2 we have not had a single missed call.  Our assumption is your engineering team must have found the root cause and did an unannounced system upgrade on 7/2.  Either that or the system restart on 7/2 corrected some still undetected system processing error.