Epicor Prophet 21 Performance – Real-World Issues
Recently, I met with an Epicor Prophet 21 customer on a discovery call to review the issues they were encountering in relation to some ongoing P21 web UI slowdowns. ERP system performance is a common challenge across the ERP community, and in the Prophet 21 community, the subject of P21 performance is similarly of great importance. Coming out of the call, I thought I’d collect a few of the talking points and add a few additional P21 system performance considerations that can impact the speed and responsiveness of your Prophet 21 web UI.
Epicor Prophet 21 system performance can be a maze to navigate.
We had originally characterized the issue as a problem with the P21 API loading, and we began looking more broadly. As you might know, the Prophet 21 sits on top of Microsoft’s Internet Information Services web server platform, known colloquially as “IIS”. There are several things to consider if your P21 web server is slowing down throughout the day, and with an ERP system like P21, the issues actually affecting the performance of the Prophet 21 web interface may reside many layers below the P21 web server.
Background:
It might be helpful to initially review the composition and operation of websites. Websites are comprised of both static and dynamic pages. A static page is pre-defined on the web server and is ready to be served up. A dynamic page is generated at run time and may dynamically differ each time it is generated. In terms of HTML pages that comprise the P21 user interface, generally speaking, the P21 application pool can only respond to a certain number of requests at a time. If it is busy responding to requests for dynamic pages, then it may not have any threads left to serve the static pages. For this reason, a code problem on a dynamic page can create the illusion that the static pages are being served “slowly”. My point is, don’t rule out code or SQL. As an example, if you have 100 pages all hitting a database or API at the same time, and all 100 await a response, request 101 may be blocked until one of the first 100 requests completes.
Diagnosing the Degradation:
Beyond explicit issues like request load, there are plenty of things that you can do to help you diagnose performance problems with your Prophet 21 web application:
Load Profiles: What does your load profile look like normally? This makes a big difference – it may be that you always have an issue, but you can’t see the impact until your site receives load. You could try to test this (in staging) with something like JMeter.
Reviewing your logs: Does your application have logs? If not, you should consider adding some logging. If you already have logs, what do they say? Are there exceptions being thrown by your application? Is there something that is consistently failing?
IIS Logs: Enable IIS logs if you haven’t already. Reviewing your P21 IIS logs can help you see which requests are taking the longest. You can use something like Microsoft’s Log Parser to run SQL-like queries against your logs. You may even want to dump your logs into a SQL database if that makes your P21 logs easier to review. Once you know which pages are taking the longest, you can focus some of your attention on them.
Memory: How much memory is your application pool using? A memory leak is an obvious candidate but should be quite easy to see. Use Windows’ inbuilt Performance Monitor to track memory consumed by your application pool over the day and see if this increases as the day goes on.
SQL Performance: The performance of your P21 SQL database may be an underlying cause of poor Prophet 21 user interface performance. SQL server provides a series of query structures called Dynamic Management Views, or DMVs, that can provide details about server and database health and performance. These can be very helpful in diagnosing performance issues at this level. One common DMV, sys.dm_exec_requests, can help you understand query properties such as wait_type, wait_time, blocking_session_id and the total_elapsed_time.
P21 Application Pool Connections: Check how many connections your application pool has open – using something like Microsoft’s TCPView. Your application pool will try to re-use connections where possible, but you’ll probably see a lot of open connections to your application pool. One interesting thing you can see from this is how many connections you have open to your SQL database or any external APIs your application is using.
Use an Application Performance and Monitoring Tool: Performance monitoring tools, like AppDynamics, will be able to help pinpoint slow performing parts of your code. Unfortunately, there’s a little bit of a learning curve to be able to use these tools effectively, but they can be very powerful in helping to diagnose problems with your applications.
SQL Server AutoGrowth Property: Review the property in your SQL database pertaining to AutoGrowth. You may encounter issues if the following are occurring:
1. If the database is a super-busy database, transactionally speaking.
2. If AutoGrowth is enabled.
3. The AutoGrowth default is a smaller MB amount. This may cause random slowdowns on the database engine, which could impact the API application pool response time.
One thing to test would be to set that AutoGrowth size in MB to a very large number. That way, the AutoGrowth will only happen periodically.
Look for Memory Leaks: Once I had a customer experiencing IIS performance degradation issues with a custom web application we had built that was using asp.net and Crystal runtime integration. Ultimately, the issues with IIS and the web app related to memory leaks that were not obvious at all until we started doing some deep-dive testing. You will want to consider the possibility of internal memory leaks when building a support case against the application having performance-related issues that may or may not have been resolved in minor version changes. I know IIS also plays a part in this and how it manages internal garbage disposal with application pools, so this may be an area that you need to explore as well.
As you can see, Epicor Prophet 21 system performance can be a maze to navigate. To find your way through the P21 performance maze, there are many potential paths to take, and depending on the ultimate source of the problem, many might be dead ends. But in understanding the directions one might take in navigating the many potential Prophet 21 performance issues, P21 users can hopefully find themselves at the maze’s end – and moving on to bigger and better things.
