Web service unexpectedly hangs

[marco.pozzi]

We're using LogicalDOC 8.0, but we had the same problem with previous 7.x versions.

After a few days of normal activity, the web service hangs and Windows service must be restarted. We have only 2 active users that connect by web frontend and Outlook addin.

Apparently, there is no info in the logs.

What can I do to further investigate on this problem?

Thanks

Marco

[car031]

look at the log file repository/logs/dms.log

Check if this happens when the indexing is running. I is really CPU intensive.

[marco.pozzi]

dms.log

[marco.pozzi]

It happened again last Friday.

I'm attaching current dms.log but I don't know if it can help.

tomcat8.exe process on the server is quite CPU intensive (> 40%).

I'm going to restart the service now. Please let me know what I can do.

Thanks

[marco.pozzi]

I'm attaching also dms.log after restarting the service.

[car031]

the log appears ok, isn't it the Document Indexer task running when yo experience that issue ?

Have you tried to increase the memory assigned to tomcat (see the attached file)

[marco.pozzi]

No, I didn't find the Document Indexer task runnning when the server hangs.

I increased the memory assigned to Tomcat to 2 Gb (it was 900 Mb). What do you suggest about memory sizing? The system is running with just 3 users.

Is there any memory leak prevention technique that I can adopt?

Thanks

Marco

[car031]

if you have enough RAM, assing 4.5 GB to LogicalDOC. No memory leak, but be aware that when you transfer files through the webservice, the full size of the file is loaded in the RAM before being flushed in the storage so you may have memory usage peaks.

[marco.pozzi]

Dear support,
the web service has crashed again. The process tomcat8.exe reports a working set peak of about 820 Mb which is much lower than the maximum memory pool (2 Gb).

So I'm thinking that this problem is not related to memory.

What else can I check on the server?

Thanks

Marco

[car031]

i cannot say for sure what your issue is, i imagine you can find more informations in the logs folder tomcat/logs/*

But the size you grant to Java with -Xmx does not refer to the tomcat process footprint but to the Java heap. You should tryu to give more and more memory before assuming this will not solve, try directly 4.5GB

[marco.pozzi]

Thanks, I increased the memory to 4500 Mb and attached the logs.

Could you please take a look to these logs?

Thanks

Marco

[car031]

In the log there is nothing that leads to memory problem. But does the tomcat process simply stops ?

How big is the document you are uploading?

[marco.pozzi]

The whole archive contains about 8.000 documents (mostly emails with small attachments, < 2 Mb). The upload is normally done via Outlook plugin.

The last time it crashed a user was uploading a file of about 300 Kb.

I think that this problem could be related to multiple opened sessions that are not correctly handled by the service.

Marco

[marco.pozzi]

Dear support,
service stopped responding again, even with 4500 Mb assigned memory.

tomcat8.exe process is about 800 Mb.

%CPU continues to vary between 12 and 27.

Clients don't get any reply from the web service, but http port is still open on the server.

Do you have any suggestion to further investigate on this annoying problem?

Thanks

Marco

[marco.pozzi]

Dear Support,
the problem has occured again. Do you have any suggestion?

Thanks

Marco

[marco.pozzi]

Dear Support,
this issue is still open and the problem occurred again.

Could you please help? I have to restart the service each time.

Thank you

Marco

[adminusername]

Hi Marco,
what kind of files do you usually send through the web service?
What is their size?
Have you tried to disable the Indexer Task in Administration.
What methods do you use of the SOAP API?

[marco.pozzi]

Hi, we usually upload very light files (< 3 Mb), most in .msg (Outlook mail format) via Outlook Addon.

I think that the problem is not related to indexing.

Could you please help?

Thanks

Marco

[adminusername]

It seems that your log file is not correctly configured, it makes a lot of not really useful logs.
If you can attach here we can correct it.

1) Disable all the Scheduled Tasks
2) Check your server has stable and linear performance

If you are using a virtual or cloud server you should know that very often the performance of these servers is floating. At certain times the virtual machine works perfectly, in others it has significant slowdowns

[marco.pozzi]

Replying to adminusername:

It seems that your log file is not correctly configured, it makes a lot of not really useful logs.
If you can attach here we can correct it.

1) Disable all the Scheduled Tasks
2) Check your server has stable and linear performance

If you are using a virtual or cloud server you should know that very often the performance of these servers is floating. At certain times the virtual machine works perfectly, in others it has significant slowdowns

Thanks for your reply. Today, web service stopped responding again and I rebooted the server.

I didn't disable scheduled tasks yet because I need indexing uploaded files. What do you exactly mean with "disable all the scheduled tasks"? Should I also stop indexing service?

Regarding the performance of the server, it is a virtual machine (ESXi 6.0) but only 3 users are served. Is there any special configuration setting I should consider in the setup of the virtual machine?

Finally, regarding my detailed log... how to I can correctly configure logging?

I cannot attach the whole log here because of uploaded files size limitation. Where I can send it for further analysis?

Thanks

Marco

[car031]

We suggest to stop the indexing service when you use the WebService? because of the indexing take a lot of RAM and CPU.

Performances are not driven by the number of users only: high load may be due to heavy scheduled tasks like the indexing.

You can use your preferred way to send big files, then just attach in the ticket the download link. You could use We Tansfer

[marco.pozzi]

Please take a look to the log here:
​https://drive.google.com/open?id=1UMeqZSfI5JSlXKi5Vnklu13xGlcg2Nz2

What do you suggest?

Thanks

Marco

[marco.pozzi]

Replying to car031:

We suggest to stop the indexing service when you use the WebService? because of the indexing take a lot of RAM and CPU.

Performances are not driven by the number of users only: high load may be due to heavy scheduled tasks like the indexing.

You can use your preferred way to send big files, then just attach in the ticket the download link. You could use We Tansfer

Did you take a look to the log? Do you have any further advice?

Thanks

Marco

[car031]

in the log there is nothing strange, it seems the application never went down. Moreover we do not see indications of CPU overload nor of JVM crashes, nor database connection issues.

Is LogicalDOC installed in a real server or is it a VM ? If it is in a VM, perhaps LogicalDOC does not receive CPU cycles and you could tune the VM engine.

[marco.pozzi]

Replying to car031:

in the log there is nothing strange, it seems the application never went down. Moreover we do not see indications of CPU overload nor of JVM crashes, nor database connection issues.

Is LogicalDOC installed in a real server or is it a VM ? If it is in a VM, perhaps LogicalDOC does not receive CPU cycles and you could tune the VM engine.

LogicalDOC is running on a ESXi VM. The other VMs are running fine, with no performance problems. Please notice that I have this issue since about 1 year. Before, LogicalDOc has been running with no problems. In any case, which VM settings do you suggest?

Do you have any tool to track web engine process? I suspect that the problem is related to web server.

Thanks

Marco

[car031]

we are not enough skilled on VM to give yo clear directions. Basically the VM allocate CPU times to the different VM, perhaps if the VM of LogicalDOC is asking a lot of CPU for a long time, the VM freezes LogicalDOC and come back with the CPU later. This is just ad idea, but you could investigate if you can tune this.

As per the logs it seems that the LogicalDOC application never went down, however yuou could profile the Java runtime that executes LogicalDOC in order to verify what happens under the covers, you could use VisualVM -> ​https://visualvm.github.io/download.html

From 7.x to 8.x we applied several performances optimizations so basically the 8.1 is more performant than older releases.