Speaker "Kaiqing Fan" Details Back



Optimize the Logic A Little, Shorten the Execution Time A Lot ----An Introduction to New Combination Processing Method


Today the tremendous large data files, big data files are keeping changing, it causes more and more troubles to handle these data files with higher requirements. Banking is not an exception too, among daily work, how to handle tremendous data files, and significantly decrease running time is a hot topic. We have many technical skills, parallel runs, they work very well but with short-comings, and cannot hugely shorten the execution time. We found if we optimize the logic a little bit to handle the business requirements, process the steps with a combination processing or set processing method, we can hugely decrease the execution time. It does not require additional server memory, additional money, but can significantly decrease huge amount running time for our SAS engines. The more the number of data files, the larger of them, the more time we can decrease if we can mix SAS technical skills, parallel run or multiple threads and combination processing method into SAS engine. It can decrease more than 90% of execution time. We already did it. Actually we already proved that this method does not only work in SAS language but also Python, and other computer languages. Key Words: Big Data, shorten execution time, optimization, Combination Processing Method or Set Processing Method


Sr. SAS Tech Lead, Sr. Data Scientist, Sr. SAS and R Developer, Statistician with 10 year experience in software programming, 3 year in statistical Analysis and developing SAS engines; Professional in handling big data, massive large data files, optimization of software engines, automation execution of SAS engines, hugely cutting of SAS engines' execution times (which can beat python performance on SPARK); I got my Master degree of Statistics from University of Wyoming, Master degree of Applied Mathematics from University of New Orleans, Master degree of History from East China Normal University, Bachelor degree of History from Liaochen University.