Transformations for algorithm optimization have shown to be effective in high-level synthesis. When a large number of transformations are available, it is always difficult to determine which transformations should be applied and in what order. In this report, we propose a methodology which clearly addresses these issues and organizes them in a systematic fashion. The proposed methodology is composed of a set of sub-tasks including bottleneck identification (why transformations should be applied), algorithm partitioning (which parts of an algorithm should be transformed), transformation prediction/selection (which transformations to apply), transformation ordering (the order in which the transformations are applied), and transformation execution (how to apply the selected transformations). A framework based on this methodology and aimed at the optimization of speed, area or power consumption of custom DSP designs, is under development. Assisted by such a framework, designers can easily and quickly to apply a variety of transformations to explore the algorithmic design space to reach better designs.