CADET Process Parameter Estimation

Chromatography Modeling
21

Hey Yemi,
from my point of view, the first part of the code looks solid. I ran it on my device and there was no error.

Regarding the callbacks, I want to give you some more background. When a callback is added to the optimization problem as you did in this line of code

optimization_problem.add_callback(
    callback, requires=[process_simulator], frequency= 1, keep_progress=True
)

the variable evaluation_objects is by default -1. This means all evaluation objects (process_75 and process_300) are used in the callback. So if the callback function is used, the code will iterate over all your evaluation objects and run the callback function.

This means if you specify the callback, as shown above, the code will run the first comparator plot function for the process_75 and will save the result. After that, it will continue with the next comparator where the evaluation object did not change. It is still process_75. This means you will have the same path set for saving the png. So you will just overwrite the first comparator plot.

For now, I would suggest that you use the following line of code, where you check the evaluation_object and use the right comparator.

def callback(
        simulation_results, individual, evaluation_object, callbacks_dir='./'):
    if evaluation_object.name == 'process_75':
        comparator_75.plot_comparison(
            simulation_results,
            file_name=f'{callbacks_dir}/{individual.id}_{evaluation_object}_comparison.png',
        show=False
        )
    else:
        comparator_300.plot_comparison(
            simulation_results,
            file_name=f'{callbacks_dir}/{individual.id}_{evaluation_object}_comparison.png',
            show=False
            )

Best,
Lukas

22

Hey,

as Lukas pointed out, the issue lies with the evaluation objects. However, I believe, his solution is not quite complete yet. Itā€™s also necessary to map the objectives to the corresponding processes.

optimization_problem.add_objective(
    comparator_75, n_objectives=comparator_75.n_metrics, requires=[process_simulator]
    evaluation_objects=process_75
)
optimization_problem.add_objective(
    comparator_300, n_objectives=comparator_300.n_metrics, requires=[process_simulator]
    evaluation_objects=process_300
)

Finally, I would simply define and add two different callbacks and also associate them with the evaluation objects:

def plot_75(
        simulation_results, individual, evaluation_object, callbacks_dir):
        comparator_75.plot_comparison(
            simulation_results,
            file_name=f'{callbacks_dir}/{individual.id}_{evaluation_object}_comparison.png',
            show=False
        )

optimization_problem.add_callback(
    plot_75, requires=[process_simulator],
    evaluation_objects=process_75
)

def plot_300(
        simulation_results, individual, evaluation_object, callbacks_dir):
        comparator_300.plot_comparison(
            simulation_results,
            file_name=f'{callbacks_dir}/{individual.id}_{evaluation_object}_comparison.png',
            show=False
        )

optimization_problem.add_callback(
    plot_300, requires=[process_simulator],
    evaluation_objects=process_300
)

I know it gets quite verbose but for now thatā€™s what weā€™ve got. :nerd_face:

Iā€™ve got a meeting with Lukas tomorrow, maybe he has some other ideas. Also, any feedback is always appreciated.

23

Thank you @l.thiel, @j.schmoelder for the detailed responses.

Jo, I was able to map the objectives as you pointed out including addition of the corresponding process. Also, I defined the two callbacks as you have shown. Here is the log I am getting currently.

Starting optimization
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\problem.py:285, in Problem._format_dict(self, out, N, return_values_of)
    284 try:
--> 285     v = v.reshape(shape[name])
    286 except Exception as e:

ValueError: cannot reshape array of size 420 into shape (10,21)

During handling of the above exception, another exception occurred:

Exception                                 Traceback (most recent call last)
Input In [1], in <cell line: 202>()
    202 if __name__ == '__main__':
    203     print('Starting optimization')
--> 204     optimization_results = optimizer.optimize(optimization_problem, use_checkpoint=True)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizer.py:119, in OptimizerBase.optimize(self, optimization_problem, save_results, results_directory, log_level, delete_cache, remove_similar, *args, **kwargs)
    115 plt.switch_backend('agg')
    117 start = time.time()
--> 119 self.run(optimization_problem, *args, **kwargs)
    121 self.results.time_elapsed = time.time() - start
    123 if delete_cache:

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\pymooAdapter.py:75, in PymooInterface.run(self, optimization_problem, use_checkpoint, update_parameters)
     73 n_gen = 1
     74 while self.algorithm.has_next():
---> 75     self.algorithm.next()
     77     X = self.algorithm.pop.get("X").tolist()
     78     F = self.algorithm.pop.get("F").tolist()

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\algorithm.py:161, in Algorithm.next(self)
    159 # call the advance with them after evaluation
    160 if infills is not None:
--> 161     self.evaluator.eval(self.problem, infills, algorithm=self)
    162     self.advance(infills=infills)
    164 # if the algorithm does not follow the infill-advance scheme just call advance
    165 else:

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\evaluator.py:69, in Evaluator.eval(self, problem, pop, skip_already_evaluated, evaluate_values_of, count_evals, **kwargs)
     65 # evaluate the solutions (if there are any)
     66 if len(I) > 0:
     67 
     68     # do the actual evaluation - call the sub-function to set the corresponding values to the population
---> 69     self._eval(problem, pop[I], evaluate_values_of, **kwargs)
     71 # update the function evaluation counter
     72 if count_evals:

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\evaluator.py:90, in Evaluator._eval(self, problem, pop, evaluate_values_of, **kwargs)
     87 X = pop.get("X")
     89 # call the problem to evaluate the solutions
---> 90 out = problem.evaluate(X, return_values_of=evaluate_values_of, return_as_dictionary=True, **kwargs)
     92 # for each of the attributes set it to the problem
     93 for key, val in out.items():

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\problem.py:187, in Problem.evaluate(self, X, return_values_of, return_as_dictionary, *args, **kwargs)
    184     only_single_value = not (isinstance(X, list) or isinstance(X, np.ndarray))
    186 # this is where the actual evaluation takes place
--> 187 _out = self.do(X, return_values_of, *args, **kwargs)
    189 out = {}
    190 for k, v in _out.items():
    191 
    192     # copy it to a numpy array (it might be one of jax at this point)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\problem.py:232, in Problem.do(self, X, return_values_of, *args, **kwargs)
    229     self._evaluate_vectorized(X, out, *args, **kwargs)
    231 # finally format the output dictionary
--> 232 out = self._format_dict(out, len(X), return_values_of)
    234 return out

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\problem.py:287, in Problem._format_dict(self, out, N, return_values_of)
    285                 v = v.reshape(shape[name])
    286             except Exception as e:
--> 287                 raise Exception(
    288                     f"Problem Error: {name} can not be set, expected shape {shape[name]} but provided {v.shape}", e)
    290         ret[name] = v
    292 # if some values that are necessary have not been set

Exception: ('Problem Error: F can not be set, expected shape (10, 21) but provided (10, 42)', ValueError('cannot reshape array of size 420 into shape (10,21)'))

Thanks and all the best,
Yemi

24

Hey Yemi,

thanks for reporting the error. I think Johannes has fixed this error in his commit from the fifth of September. Do you use the latest dev branch?

Have a nice day.

Best Lukas

25

I also pushed another fix. Hope it runs now.

We updated the function signature for adding optimization variables. I took the liberty to create a gist with the latest syntax.

1 Like
26

Hi Jo,
I was just able to update my dev branch build today. I modified my script based on your gist.
I am returning another ā€œis_file_classā€ error. Please see below.

Starting optimization
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\multiprocess\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\multiprocess\pool.py", line 48, in mapstar
    return list(map(*args))
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pathos\helpers\mp_helper.py", line 15, in <lambda>
    func = lambda args: f(*args)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py", line 705, in eval_fun
    results = self.evaluate_objectives(
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py", line 108, in wrapper
    return func(self, x, *args, **kwargs)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py", line 683, in evaluate_objectives
    value = self._evaluate(x, objective, force)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py", line 108, in wrapper
    return func(self, x, *args, **kwargs)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py", line 1446, in _evaluate
    result = step.evaluate(current_request)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py", line 2450, in __call__
    results = self.evaluator(request, *args, **kwargs)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\simulator\simulator.py", line 196, in simulate
    results = self.simulate_n_cycles(
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\log.py", line 82, in wrapper
    result = function(*args, **kwargs)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\log.py", line 133, in wrapper
    results = function(*args, **kwargs)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\log.py", line 112, in wrapper
    raise e
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\log.py", line 104, in wrapper
    return function(*args, **kwargs)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\simulator\simulator.py", line 250, in simulate_n_cycles
    return self.run(process, **kwargs)
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\simulator\cadetAdapter.py", line 232, in run
    if cadet.is_file:
  File "C:\Users\~\Anaconda3\envs\CADETProcessTest\lib\site-packages\cadet\cadet.py", line 169, in is_file
    if self._is_file_class is not None:
AttributeError: 'Cadet' object has no attribute '_is_file_class'
"""

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
Input In [1], in <cell line: 147>()
    147 if __name__ == '__main__':
    148     print('Starting optimization')
--> 149     optimization_results = optimizer.optimize(
    150         optimization_problem,
    151         use_checkpoint=False
    152     )

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizer.py:124, in OptimizerBase.optimize(self, optimization_problem, save_results, results_directory, log_level, delete_cache, remove_similar, *args, **kwargs)
    120 plt.switch_backend('agg')
    122 start = time.time()
--> 124 self.run(optimization_problem, *args, **kwargs)
    126 self.results.time_elapsed = time.time() - start
    128 if delete_cache:

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\pymooAdapter.py:75, in PymooInterface.run(self, optimization_problem, use_checkpoint, update_parameters)
     73 n_gen = 1
     74 while self.algorithm.has_next():
---> 75     self.algorithm.next()
     77     X = self.algorithm.pop.get("X").tolist()
     78     F = self.algorithm.pop.get("F").tolist()

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\algorithm.py:161, in Algorithm.next(self)
    159 # call the advance with them after evaluation
    160 if infills is not None:
--> 161     self.evaluator.eval(self.problem, infills, algorithm=self)
    162     self.advance(infills=infills)
    164 # if the algorithm does not follow the infill-advance scheme just call advance
    165 else:

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\evaluator.py:69, in Evaluator.eval(self, problem, pop, skip_already_evaluated, evaluate_values_of, count_evals, **kwargs)
     65 # evaluate the solutions (if there are any)
     66 if len(I) > 0:
     67 
     68     # do the actual evaluation - call the sub-function to set the corresponding values to the population
---> 69     self._eval(problem, pop[I], evaluate_values_of, **kwargs)
     71 # update the function evaluation counter
     72 if count_evals:

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\evaluator.py:90, in Evaluator._eval(self, problem, pop, evaluate_values_of, **kwargs)
     87 X = pop.get("X")
     89 # call the problem to evaluate the solutions
---> 90 out = problem.evaluate(X, return_values_of=evaluate_values_of, return_as_dictionary=True, **kwargs)
     92 # for each of the attributes set it to the problem
     93 for key, val in out.items():

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\problem.py:187, in Problem.evaluate(self, X, return_values_of, return_as_dictionary, *args, **kwargs)
    184     only_single_value = not (isinstance(X, list) or isinstance(X, np.ndarray))
    186 # this is where the actual evaluation takes place
--> 187 _out = self.do(X, return_values_of, *args, **kwargs)
    189 out = {}
    190 for k, v in _out.items():
    191 
    192     # copy it to a numpy array (it might be one of jax at this point)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\problem.py:229, in Problem.do(self, X, return_values_of, *args, **kwargs)
    227     self._evaluate_elementwise(X, out, *args, **kwargs)
    228 else:
--> 229     self._evaluate_vectorized(X, out, *args, **kwargs)
    231 # finally format the output dictionary
    232 out = self._format_dict(out, len(X), return_values_of)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pymoo\core\problem.py:237, in Problem._evaluate_vectorized(self, X, out, *args, **kwargs)
    236 def _evaluate_vectorized(self, X, out, *args, **kwargs):
--> 237     self._evaluate(X, out, *args, **kwargs)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\pymooAdapter.py:242, in PymooProblem._evaluate(self, x, out, *args, **kwargs)
    240 opt = self.optimization_problem
    241 if opt.n_objectives > 0:
--> 242     F = opt.evaluate_objectives_population(
    243         x,
    244         untransform=True,
    245         n_cores=self.n_cores,
    246     )
    247     out["F"] = np.array(F)
    249 if opt.n_nonlinear_constraints > 0:

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py:108, in OptimizationProblem.untransforms.<locals>.wrapper(self, x, untransform, *args, **kwargs)
    105 if untransform:
    106     x = self.untransform(x)
--> 108 return func(self, x, *args, **kwargs)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py:119, in OptimizationProblem.ensures2d.<locals>.wrapper(self, population, *args, **kwargs)
    116 population = np.array(population, ndmin=2)
    117 population = population.tolist()
--> 119 return func(self, population, *args, **kwargs)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py:725, in OptimizationProblem.evaluate_objectives_population(self, population, force, n_cores)
    722     self.cache.close()
    724     with pathos.pools.ProcessPool(ncpus=n_cores) as pool:
--> 725         results = pool.map(eval_fun, population)
    727 return results

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pathos\multiprocessing.py:139, in ProcessPool.map(self, f, *args, **kwds)
    137 AbstractWorkerPool._AbstractWorkerPool__map(self, f, *args, **kwds)
    138 _pool = self._serve()
--> 139 return _pool.map(star(f), zip(*args))

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\multiprocess\pool.py:364, in Pool.map(self, func, iterable, chunksize)
    359 def map(self, func, iterable, chunksize=None):
    360     '''
    361     Apply `func` to each element in `iterable`, collecting the results
    362     in a list that is returned.
    363     '''
--> 364     return self._map_async(func, iterable, mapstar, chunksize).get()

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\multiprocess\pool.py:771, in ApplyResult.get(self, timeout)
    769     return self._value
    770 else:
--> 771     raise self._value

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\multiprocess\pool.py:125, in worker()
    123 job, i, func, args, kwds = task
    124 try:
--> 125     result = (True, func(*args, **kwds))
    126 except Exception as e:
    127     if wrap_exception and func is not _helper_reraises_exception:

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\multiprocess\pool.py:48, in mapstar()
     47 def mapstar(args):
---> 48     return list(map(*args))

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\pathos\helpers\mp_helper.py:15, in starargs.<locals>.<lambda>()
     13 def starargs(f):
     14     """decorator to convert a many-arg function to a single-arg function"""
---> 15     func = lambda args: f(*args)
     16    #func.__module__ = f.__module__
     17    #func.__name__ = f.__name__
     18     doc = "\nNOTE: all inputs have been compressed into a single argument"

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py:705, in eval_fun()
    704 def eval_fun(ind):
--> 705     results = self.evaluate_objectives(
    706         ind,
    707         force=force,
    708     )
    709     self.cache.close()
    711     return results

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py:108, in wrapper()
    105 if untransform:
    106     x = self.untransform(x)
--> 108 return func(self, x, *args, **kwargs)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py:683, in evaluate_objectives()
    681 for objective in self.objectives:
    682     try:
--> 683         value = self._evaluate(x, objective, force)
    684         f += value
    685     except CADETProcessError:

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py:108, in wrapper()
    105 if untransform:
    106     x = self.untransform(x)
--> 108 return func(self, x, *args, **kwargs)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py:1446, in _evaluate()
   1444     step.evaluate(current_request, eval_obj)
   1445 else:
-> 1446     result = step.evaluate(current_request)
   1447     if step not in self.cached_steps:
   1448         tag = 'temp'

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\optimization\optimizationProblem.py:2450, in __call__()
   2447 else:
   2448     kwargs = self.kwargs
-> 2450 results = self.evaluator(request, *args, **kwargs)
   2452 return results

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\simulator\simulator.py:196, in simulate()
    194 process.lock = True
    195 if not self.evaluate_stationarity:
--> 196     results = self.simulate_n_cycles(
    197         process, self.n_cycles, previous_results, **kwargs
    198     )
    199 else:
    200     results = self.simulate_to_stationarity(
    201         process, previous_results, **kwargs
    202     )

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\log.py:82, in wrapper()
     79 @wraps(function)
     80 def wrapper(*args, **kwargs):
     81     start = time.time()
---> 82     result = function(*args, **kwargs)
     83     elapsed = time.time() - start
     84     logger = get_logger(logger_name, level=None)

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\log.py:133, in wrapper()
    129 logger = get_logger(logger_name, level=None)
    131 logger.debug('{} was called with {}, {}'.format(
    132         function, *args, **kwargs))
--> 133 results = function(*args, **kwargs)
    134 logger.debug(f'Results: {results}')
    136 return results

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\log.py:112, in wrapper()
    109 logger.exception(err)
    111 # re-raise the exception
--> 112 raise e

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\log.py:104, in wrapper()
    102 logger = get_logger(logger_name, level=None)
    103 try:
--> 104     return function(*args, **kwargs)
    105 except Exception as e:
    106     # log the exception
    107     err = "There was an exception in "

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\simulator\simulator.py:250, in simulate_n_cycles()
    247 if previous_results is not None:
    248     self.set_state_from_results(process, previous_results)
--> 250 return self.run(process, **kwargs)
    252 self.n_cycles = n_cyc_orig

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\CADETProcess\simulator\cadetAdapter.py:232, in run()
    228     cadet = CadetAPI()
    230 cadet.root = self.get_process_config(process)
--> 232 if cadet.is_file:
    233     if file_path is None:
    234         cadet.filename = self.get_tempfile_name()

File ~\Anaconda3\envs\CADETProcessTest\lib\site-packages\cadet\cadet.py:169, in is_file()
    167 if self._is_file is not None:
    168     return bool(self._is_file)
--> 169 if self._is_file_class is not None:
    170     return bool(self._is_file_class)

AttributeError: 'Cadet' object has no attribute '_is_file_class'
27

Hey, are you sure that CADET is installed properly in your environment?

If so, we could also have a short call to figure out the issue together.

28

I believe so. I havenā€™t changed anything in the environment except to install the recent CADET-Process-dev build.
Can you send me a zoom invite to my email for whenever youā€™re available? Iā€™ll send you my email privately.

29

Hi @j.schmoelder,
First, thanks to you Jo for all the offline help with getting my simulations and optimizations going.
Iā€™m slowly gaining more understanding of how CADET-Process works and how to setup my optimization problem within my chromatography system.

In my current setup, I have set my flowsheet to include the system dead volume and the volume behind column with a tubular reactor and a cstr, respectively and that seems to simulate just fine.

However, in my parameter estimation, I am now getting a so-called ā€œKey Errorā€. So far I keep getting this error well into the optimization. In some cases after the 5th generation and sometimes after the 17/18th. I have a feeling this might have something to do with the upper/lower boundaries of my parameters, however, not too sure.

Please see the error log below (Iā€™ve also saved a copy of the script on the shared labs site).
Thanks,
-Yemi

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In [7], line 68
     66 if __name__ == '__main__':
     67     print('Starting optimization')
---> 68     optimization_results = optimizer.optimize(
     69         optimization_problem,
     70         use_checkpoint=False
     71     )

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\optimization\optimizer.py:124, in OptimizerBase.optimize(self, optimization_problem, save_results, results_directory, log_level, delete_cache, remove_similar, *args, **kwargs)
    120 plt.switch_backend('agg')
    122 start = time.time()
--> 124 self.run(optimization_problem, *args, **kwargs)
    126 self.results.time_elapsed = time.time() - start
    128 if delete_cache:

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\optimization\pymooAdapter.py:85, in PymooInterface.run(self, optimization_problem, use_checkpoint, update_parameters)
     82     G = len(X)*[None]
     83 X_opt = self.algorithm.opt.get("X")
---> 85 self.run_post_generation_processing(X, F, G, n_gen, X_opt)
     87 with open(checkpoint_path, "wb") as dill_file:
     88     self.algorithm.random_state = random.getstate()

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\optimization\optimizer.py:311, in OptimizerBase.run_post_generation_processing(self, X, F, G, current_generation, X_opt)
    309 pareto_front = Population()
    310 for x in X_opt:
--> 311     ind = self.results.population_all[x]
    312     pareto_front.add_individual(ind)
    314 self.results.pareto_front = pareto_front

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\optimization\population.py:426, in Population.__getitem__(self, x)
    424 def __getitem__(self, x):
    425     x = tuple(x)
--> 426     return self._individuals[x]

KeyError: (0.7421408821046899, 2.146574537970934e-06, 0.036626487084130493, 0.5606081066991573)
30

Hey Yemi,
Iā€™m traveling this week but next week Iā€™ll have a look. It seems to be an issue with the internal caching of (intermediate) results.
Best
Jo

1 Like
31

Hi Jo,
Following up on this, if you have had some time to explore it. In the interim, it appears after setting use_checkpoint to true that I am able to run past this KeyError issue, since the optimization just continues from the generations before the crash. Itā€™s just that the estimations take way longer because of it.

On the other hand, I began using the GRM model in cadet-process and I am getting an error that appears to be related to the surface_diffusion and q parameters. When I view the output of my column.parameters, it says 'surface_diffusion': None, even though I have a value set there. And the error states

CADETProcessError: CADET Error: Simulation failed with bā€™ERROR: INIT_Q does not contain enough values for all bound states\r\nā€™

Please see below for the full error log. I canā€™t figure out if Iā€™m entering values incorrectly on my end or if something else is at play here.
Best,
-Yemi

---------------------------------------------------------------------------
CADETProcessError                         Traceback (most recent call last)
Cell In [2], line 91
     87 process.add_event(
     88     'feed_on', 'flow_sheet.feed.flow_rate', flow_rate, 0
     89 )
     90 process_simulator = Cadet()
---> 91 simulation_results = process_simulator.simulate(process)
     92 _ = simulation_results.solution.tubing_outlet.outlet.plot()

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\simulator\simulator.py:196, in SimulatorBase.simulate(self, process, previous_results, **kwargs)
    194 process.lock = True
    195 if not self.evaluate_stationarity:
--> 196     results = self.simulate_n_cycles(
    197         process, self.n_cycles, previous_results, **kwargs
    198     )
    199 else:
    200     results = self.simulate_to_stationarity(
    201         process, previous_results, **kwargs
    202     )

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\log.py:82, in log_time.<locals>.log_time_decorator.<locals>.wrapper(*args, **kwargs)
     79 @wraps(function)
     80 def wrapper(*args, **kwargs):
     81     start = time.time()
---> 82     result = function(*args, **kwargs)
     83     elapsed = time.time() - start
     84     logger = get_logger(logger_name, level=None)

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\log.py:133, in log_results.<locals>.log_results_decorator.<locals>.wrapper(*args, **kwargs)
    129 logger = get_logger(logger_name, level=None)
    131 logger.debug('{} was called with {}, {}'.format(
    132         function, *args, **kwargs))
--> 133 results = function(*args, **kwargs)
    134 logger.debug(f'Results: {results}')
    136 return results

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\log.py:112, in log_exceptions.<locals>.log_exception_decorator.<locals>.wrapper(*args, **kwargs)
    109 logger.exception(err)
    111 # re-raise the exception
--> 112 raise e

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\log.py:104, in log_exceptions.<locals>.log_exception_decorator.<locals>.wrapper(*args, **kwargs)
    102 logger = get_logger(logger_name, level=None)
    103 try:
--> 104     return function(*args, **kwargs)
    105 except Exception as e:
    106     # log the exception
    107     err = "There was an exception in "

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\simulator\simulator.py:250, in SimulatorBase.simulate_n_cycles(self, process, n_cyc, previous_results, **kwargs)
    247 if previous_results is not None:
    248     self.set_state_from_results(process, previous_results)
--> 250 return self.run(process, **kwargs)
    252 self.n_cycles = n_cyc_orig

File ~\Anaconda3\envs\cadetProcess\lib\site-packages\CADETProcess\simulator\cadetAdapter.py:254, in Cadet.run(self, process, cadet, file_path)
    249 if return_information.returncode != 0:
    250     self.logger.error(
    251         f'Simulation of {process.name} '
    252         f'with parameters {process.config} failed.'
    253     )
--> 254     raise CADETProcessError(
    255         f'CADET Error: Simulation failed with {return_information.stderr}'
    256     ) from None
    258 try:
    259     results = self.get_simulation_results(
    260         process, cadet, elapsed, return_information
    261     )

CADETProcessError: CADET Error: Simulation failed with b'ERROR: INIT_Q does not contain enough values for all bound states\r\n'
32

Hey Yemi,

sorry for the delay. I caught Corona on my travels, but Iā€™m better now.

Should be fixed now. I have some internal checking procedures for parameters and I forgot to also update the parameters dict when I added some extra checks. This should, however, not have had any effects on running the simulation. It would still use the correct value.

What binding model did you use?

33

Oh man! Sorry to hear that and glad youā€™re better now.

This one was with the langmuir binding model.

34

Hey @y_ajayi,

I think we found the issue with the dying kernel. See this discussion.

Could you provide me with more information about the other issues youā€™re experiencing?

35

That sounds good.
Iā€™m glad the source of that issue has been found.
Regarding the other problems I reported above, I havenā€™t experienced them much. However, I am currently on another project for the next few months so when I come back to this (perhaps with an updated version of CADET-Process), we can revisit?

Thanks again.

1 Like