Memory Issues in C++

I am having run-time memory allocation errors with a C++ application. I have eliminated memory leaks, invalid pointer references and out-of-bounds vector assignments as the source of the issue - I am pretty sure it's something to do with memory fragmentation and I am hoping to get help with how to further diagnose and correct the problem.

My code is too large to post (about 15,000 lines - I know not huge but clearly too large to put online), so I am going to describe things with a few relevant snippets of code.

Basically, my program takes a bunch of string and numerical data sets as inputs (class objects with vector variables of type double, string, int and bool), performs a series of calculations, and then spits out the resulting numbers. I have tested and re-tested the calculations and outputs - everything is calculating as it should, and on smaller datasets things run perfectly.

However, when I scale things up, I start getting memory allocation errors, but I don't think I am even close to approaching the memory limits of my system - please see the two graphs below...my program cycles through a series of scenarios (performing identical calculations under a different set of parameters for each scenario) - in the first graph, I run 7 scenarios on a dataset of about 200 entries. As the graph shows, each "cycle" results in memory swinging up and back down to its baseline, and the overall memory usage is tiny (see the seven small blips on the right half of the bottom graph). On the second graph, I am now running a dataset of about 10,000 entries (see notes on dataset below). In this case, I only get through 2 full cycles before getting my error (as it is trying to resize a class object for the third scenario). You can see the first two scenarios in the bottom right-half graph; a lot more memory usage than before, but still only a small fraction of available memory. And as with the smaller dataset, usage increases while my scenario runs, and then decreases back to it's initial level before reaching the next scenario.

This pattern, along with other tests I have done, lead me to believe it's some sort of fragmentation problem. The error always occurs when I am attempting to resize a vector, although the particular resize operation that causes the error varies based on the dataset size. Can anyone help me understand what's going on here and how I might fix it? I can describe things in much greater detail but already felt like my post was getting long...please ask questions if you need to and I will respond/edit promptly.

Clarification on the data set The numbers 200 and 10,000 represent the number of unique records I am analyzing. Each record contains somewhere between 75 and 200 elements / variables, many of which are then being manipulated. Further, each variable is being manipulated over time and across multiple iterations (both dimensions variable). As a result, for an average "record" (the 200 to 10,000 referenced above), there could be easily as many as 200,000 values associated with it - a sample calculation:

1 Record * 75 Variables * 150 periods * 20 iterations = 225,000 unique values per record.

Offending Code (in this specific instance):

vector<LoanOverrides> LO;
LO.resize(NumOverrides + 1); // Error is occuring here.  I am certain that NumOverrides is a valid numerical entry = 2985

// Sample class definition
class LoanOverrides {
public:
  string IntexDealName;
  string LoanID;
  string UniqueID;
  string PrepayRate;
  string PrepayUnits;
  double DefaultRate;
  string DefaultUnits;
  double SeverityRate;
  string SeverityUnits;
  double DefaultAdvP;
  double DefaultAdvI;
  double RecoveryLag;
  string RateModRate;
  string RateModUnits;
  string BalanceForgivenessRate;
  string BalanceForgivenessRateUnits;
  string ForbearanceRate;
  string ForbearanceRateUnits;
  double ForbearanceRecoveryRate;
  string ForbearanceRecoveryUnits;
  double BalloonExtension;
  double ExtendPctOfPrincipal;
  double CouponStepUp;
};

---

You have a 64-bit operating system capable of allocating large quantities of memory, but have built your application as a 32-bit application, which can only allocate a maximum of about 3GB of memory. You are trying to allocate more than that.

Try compiling as a 64-bit application. This may enable you to reach your goals. You may have to increase your pagefile size.

See if you can dispose of intermediate results earlier than you are currently doing so.

Try calculating how much memory is being used/would be used by your algorithm, and try reworking your algorithm to use less.

Try avoiding duplicating data by reworking your algorithm. I see you have a lot of reference data, which by the looks of it isn't going to change during the application run. You could put all of that into a single vector, which you allocate once, then refer to them via integer indexes everywhere else, rather than copying them. (Just guessing that you are copying them).

Try avoiding loading all the data at once by reworking your algorithm to work on batches.

Without knowing more about your application, it is impossible to offer better advice. But basically you are running out of memory because you are allocating a huge, huge amount of it, and based on your application and the snippets you have posted I think you can probably avoid doing so with a little thought. Good luck.

链接地址: http://www.djcxy.com/p/63274.html

上一篇: 身份验证和我的身份验证

下一篇: C ++中的内存问题