Cross-country analysis of the effectiveness of commercialization of scientific research results

The objective of this research is to undertake a comparative analysis of methodologies for evaluating the efficacy of research commercialization across diverse nations, focusing particularly on the United States, Australia, Canada, South Korea, the European Union, and Kazakhstan. The study methodology involved a comparative analysis, commencing with an exhaustive examination of academic and practical resources to pinpoint key organizations involved in research commercialization within the specified countries. Subsequently, after the selection of countries and a systematic scrutiny of these organizations' activities, the methodology endorsed by the Alliance of Technology Transfer Professionals (ATTP) was employed to compare and assess commercialization efficiency across varied nations. Results of the study: the study outcomes unveiled common trends and effective assessment approaches, while also identifying deficiencies in Kazakhstan's commercialization evaluation system and offering recommendations for enhancement. The assessment of commercialization metrics in Kazakhstan is hindered by the paucity of comprehensive data, rendering comparisons with global benchmarks, including the ATTP methodology, challenging. However, concerted collaboration among governmental bodies, research institutions, and industrial stakeholders could surmount these hurdles and foster innovative entrepreneurship within the nation. The adoption of alternative research methodologies such as environmental functioning analysis models or regression analysis may enable a more profound evaluation of commercialization efficiency in Kazakhstan, notwithstanding data constraints


Introduction
The commercialization of research results has become an increasingly relevant topic among enterprises across various industries, as well as government and non-governmental organizations worldwide in recent years (Kumar et al., 2015).Its goals extend far beyond merely generating profit from selling scientific ideas.The commercialization of scientific developments can address technological, economic, environmental, and social issues in both developed and developing countries (Gibson, 2014), thereby positively impacting company profits, increasing tax revenues, and enhancing global and national competitiveness (Sutopo et al., 2019).However, it should be noted that transferring innovative ideas from the research laboratory through production, marketing, and sales to the customer is a complex and risky task (Sung, 2009).
In many cases, many technological products resulting from research fail to reach the market due to technology readiness issues (Aoshima et al., 2011), lack of skills (Kusuma et al., 2015), absence of commercialization vision and mindset in R&D (Karaveg et al., 2014), among many other reasons.Therefore, it is crucial to study critical factors and find ways for effective technology commercialization (Sung, 2009).This will allow assessing the commercialization potential of R&D at various stages, adjusting strategies, and preventing resource diversion to unpromising projects (Karaveg et al., 2014).Moreover, timely assessment of R&D commercialization projects' effectiveness will help select potential projects with high commercial value.
The works of Alibekova (2018) and Suleimenova ( 2018) discuss domestic research on the evaluation of R&D commercialization effectiveness in Kazakhstan, highlighting the relevance of this topic at the local level.The lack of comparative studies on the evaluation of R&D commercialization effectiveness across countries further justifies the need for this research.
Currently, the commercialization of research varies significantly across countries depending on factors such as the economic environment, the level of R&D infrastructure, government policies, and cultural attitudes towards entrepreneurship and innovation.
Globally, the assessment of research commercialization is usually conducted at different levels: universities and research institutes typically conduct internal evaluations, while government agencies and funding organizations often perform broader evaluations on a national or regional scale.In countries like Germany, the USA, Japan, and Canada, where commercialization and technology transfer are priorities, commercialization offices evaluate their commercialization efforts.Evaluations are generally carried out through financial indicators (return on investment), technology transfer indicators (patent monitoring), and socio-economic indicators (number of jobs created and environmental impact).However, their evaluation is a challenging task due to the complexity of measurement.
In Kazakhstan, the National Center for State Scientific and Technical Expertise (NCSTE) collects and reviews applications for grant funding for technology commercialization.The National Science Foundation (NSF) acts as the national grant operator, forming competition documentation and setting requirements for applicants and project outcomes.
The commercialization of research and technologies in universities worldwide began with the passage of the Bayh-Dole Act in the United States in 1980.However, the effectiveness assessment mechanism for commercialization systems is by no means uniform across research institutions, industries, and countries.Furthermore, the accumulated experience of various approaches to evaluating R&D and their commercialization is not exhaustive.Therefore, a comprehensive approach and analysis of different models used in various countries are required.
Thus, the aim of this research is a comparative analysis of the effectiveness of R&D commercialization in Kazakhstan and other countries.To achieve this goal, the study conducts a cross-country comparative analysis of the effectiveness of commercialization project implementation in the United States, Australia, Canada, the European Union, South Korea, and Kazakhstan.
In Kazakhstan, there is virtually no scientific research database on this topic.Domestic researchers have identified problems in evaluating the effectiveness of scientific developments' commercialization (Alibekova et al., 2018).It is proposed to transfer the evaluation of commercialization programs' effectiveness to a permanent and systematic basis, develop and apply behavioral additionality indicators (process additionality), and involve local authorized bodies in evaluating programs' effectiveness at the regional and local levels.
Overall, the study consists of the following main sections: The first part compares the frameworks for evaluating research commercialization in different countries, highlighting the various criteria for assessing commercialization effectiveness and resource differences.
The second part shifts to a comparative analysis of R&D commercialization effectiveness using standardized metrics developed by the Alliance of Technology Transfer Professionals (ATTP) to evaluate commercialization efforts.
Finally, the study contains recommendations for further research, emphasizing the importance of addressing data gaps and strengthening cooperation between government agencies, scientific institutions, and industry representatives.
Financial metrics provide insight into profitability and the overall financial health of the commercialization process, evaluated through indicators such as return on investment (ROI), profit margin, payback period, and others.These metrics allow for a quantitative assessment and the selection of technologies with the most successful parameters but do not account for the long-term and broad effects of commercialization.
Market metrics collectively provide a comprehensive picture of the commercialization process, covering market penetration, customer acquisition and retention, brand perception, overall market competitiveness, customer loyalty levels, market share, and more (Jobber & Ellis-Chadwick, 2019).However, data on these parameters are often not widely available, making it difficult to assess effectiveness using these metrics.
Innovation metrics are crucial for evaluating the success of bringing new products or services to market (Cooper, 2008) and are assessed by the number of patents, R&D investments, product revenues, and the speed of innovation implementation, with the highest indicators suggesting high commercialization effectiveness of R&D (Rogers, 2003).These metrics are most commonly used in evaluating the effectiveness of domestic R&D commercialization.
It is worth noting that customer satisfaction and feedback metrics are critical for assessing commercialization effectiveness because they provide insights into how well a product or service meets customer expectations (Dixon et al., 2010).Key parameters include consumer satisfaction levels, customer effort scores, churn rates, and customer lifetime value.Like financial metrics, these metrics can provide a quantitative assessment of commercialization effectiveness, potentially yielding more accurate and reliable data.
Social and environmental impact metrics comprehensively evaluate the effectiveness of commercialization efforts by considering not only financial success but also the broader impact on society and the environment, where less impact indicates high R&D commercialization effectiveness (Porter & Kramer, 2006).
Competitive analysis metrics, though among the least frequently used, are essential for evaluating commercialization effectiveness as they provide insights into how a company's performance and strategies compare to those of its competitors (Chaffey & Smith, 2017).The infrequent use of these metrics is due to the lack of resources, skills, and access to data needed to study competitors' indicators.
Using the aforementioned metrics individually does not provide a complete picture of R&D commercialization effectiveness; therefore, they should be used in combination.A limitation of this approach is the presence of a large number of indicators from various fields, complicating the evaluation process.
In domestic studies, quantitative evaluation methods have not been used.For evaluating the effectiveness of R&D commercialization, target indicators such as the number of patents, signed license agreements, startups, and publications were proposed based on the study of the Technology Commercialization Program by the World Bank and the Ministry of Education and Science of the Republic of Kazakhstan (Alibekova, 2018).The author noted the need to develop indicators of the long-term impact of various government and industry programs based on a systematic approach to developing comprehensive effectiveness evaluation indicators.
Therefore, the lack of comparative studies on the evaluation of R&D commercialization effectiveness across countries further justifies the need for this research.

Methodology
A systematic approach, based on a review of academic and practical sources, was employed in this study.The research can be divided into the following stages: Stage 1: An extensive analysis of academic articles, organizational reports, and other publications containing information about structures supporting and transferring scientific research in various countries was conducted.Key organizations in each of the countries under consideration were identified, such as the Association of University Technology Managers (AUTM) in the United States and Knowledge Commercialization Australasia (KCA) in Australia.The selection of countries was influenced by several factors.Firstly, the countries considered, as well as the European Union, have successful practices in the commercialization of scientific research and represent different approaches to this process.For example, the United States is a leader in global innovation and commercialization of scientific research, while Australia, Canada, and South Korea also have significant experience in this field, albeit with some differences in approaches.Additionally, the selected countries represent various regions of the world, including North America, Australia, Asia, and Europe.
Stage 2: A detailed analysis of the activities of each identified organization was conducted, including their goals, methods of operation, and practices in the commercialization of scientific research.
Stage 3: A comparative analysis of the functioning of R&D commercialization systems in various countries, including the United States, Canada, and Australia, was conducted using standardized metrics from the Alliance of Technology Transfer Professionals (ATTP).Common trends and differences in the approaches to evaluating the effectiveness of commercialization in the countries under consideration were identified.
Stage 4: Based on the analysis conducted, conclusions were drawn about the most common and successful methods for evaluating the effectiveness of research commercialization in various countries.Additionally, potential directions for further research in this area were discussed.

I. International Practices for Evaluating R&D Commercialization
Research on the effectiveness of commercialization in different countries highlights a variety of organizations dedicated to supporting and transferring scientific research.For example, in the United States, the Association of University Technology Managers (AUTM) plays a key role.In Canada, significant organizations include the National Sciences and Engineering Research Council (NSERC) and the Centers of Excellence for Commercialization of Research (CECR).In Australia, Commercialisation Australasia (KCA) operates, while in South Korea, the Korea Advanced Institute of Science & Technology (KAIST) is prominent.In Europe, the leading knowledge transfer organization is the Association of Knowledge Transfer Professionals (ASTP), which aims to share best practices and develop competencies in knowledge transfer activities.
An analysis of practices in foreign countries demonstrates that specific organizations are endowed with the authority to evaluate the effectiveness of R&D commercialization results, each practicing their criteria for assessing the effectiveness of scientific results commercialization (Tab.1).The presence of well-established technology commercialization management systems in these countries primarily indicates that the transfer of new knowledge to the real sector is a priority area of economic development.This conclusion is supported by evaluations from international organizations.According to the Global Innovation Index 2023, the United States ranked 3rd, South Korea 10th, Canada 15th, and Australia 24th.
In Kazakhstan, there is no specialized organization responsible for evaluating and monitoring the effectiveness of R&D commercialization.The Bureau of National Statistics under the Agency for Strategic Planning and Reforms of the Republic of Kazakhstan is the primary source of statistical information on science and innovation development.Data collection is conducted according to international standards, based on the Frascati Manual (for science data) and the Oslo Manual (for innovation data).Specifically, data is collected on indicators such as: - Two organizations could potentially assume the role of evaluating the effectiveness of R&D commercialization in the country: the National Center for State Scientific and Technical Expertise (NCSTE) and the Alliance of Technology Commercialization Professionals (ATCP).NCSTE is a leading organization in the field of scientific and technical information, and it collects information on the results of R&D conducted in the country.However, to assume this role, NCSTE would need to establish a dedicated division and strengthen its staffing base.ATCP, established in December 2015, is a public association of individuals and legal entities created to develop the technology commercialization system in Kazakhstan.ATCP organizes occasional training sessions and workshops in technology commercialization, provides independent comprehensive evaluations of technological projects, but does not evaluate commercialization effectiveness.To assume the role of an organization authorized to evaluate and monitor the effectiveness of commercialization processes in the country, ATCP would require resource support and official authority.
Thus, the conducted analysis highlights the diversity of research commercialization evaluation systems across countries.While countries such as the United States, Canada, Australia, South Korea, and EU countries have resources and authorized organizations, such as AUTM, NSERC, CECR, KCA, KAIST, JRC, and ASTP, actively contributing to the evaluation and improvement of research and development commercialization processes, Kazakhstan, as a developing country, is still at the planning stage of establishing an evaluation and monitoring system for the effectiveness of research commercialization.

II. Cross-Country Evaluation of R&D Commercialization Results
The Alliance of Technology Transfer Professionals (ATTP) is a leading global professional organization in research commercialization, knowledge exchange, and technology transfer.ATTP represents an alliance of over 20,000 individuals through a consortium of 14 international professional associations.ATTP's primary task is to establish standards and best practices and to confer the globally recognized designation of "Registered Technology Transfer Professional" (RTTP).
The recent ATTP report "Summary of Selected Global Commercialization Metrics: 2004-2019" provides a valuable resource for comparing commercialization efforts across countries.Although the report primarily focuses on data from countries such as the United States, Canada, Australia, Israel, the United Kingdom, South Africa, Ireland, and Japan, its standardized metrics can also be applied to Kazakhstan.In this section, we focus on the United States, Canada, and Australia, comparing them with data from Kazakhstan.
The selected metrics for this international comparison project are not exhaustive but are the most accessible.They cover various aspects, including the effectiveness of scientific projects as a precursor to research commercialization, the volume and efficiency of technology transfer, and the creation of new spin-off companies.
1) Technology Commercialization Office Resources: This metric quantitatively evaluates the average number of personnel per office engaged in knowledge exchange, technology transfer, or research commercialization.Tab. 2 shows that this indicator steadily increased in the United States over a long period (2004)(2005)(2006)(2007)(2008)(2009)(2010)(2011)(2012)(2013)(2014), while Canada and Australia experienced periods of decline and growth.It is reasonable to assume that a higher value of this metric indicates greater potential for effective commercialization opportunities.Unfortunately, detailed data on the number of full-time equivalent personnel engaged in commercialization activities per institution in Kazakhstan is currently unavailable (or does not exist).The available metric pertains only to the number of employees engaged in R&D in the country.
2) Intellectual Property Disclosure: Intellectual property (IP) is a confidential description of an invention or creative activity outcome, presented by inventors to initiate the protection and commercialization process.Specifically, according to the Civil Code of the Republic of Kazakhstan, intellectual property includes the results of intellectual creative activity and means of individualization equated to them, such as legal entity names, trademarks, service marks, etc.The "Intellectual Property Disclosure" metric provides a quantitative representation of scientific results whose information has been disclosed to the technology transfer office to facilitate their dissemination through knowledge exchange channels.This disclosure serves as an important resource for knowledge sharing.To assess the effectiveness of the R&D commercialization system, ATTP uses the metric of intellectual property disclosure activity, specifically disclosures of inventions per 100 million USD of R&D expenditure for Australia, Canada, and the United States (Tab.3).NCSTI will provide the figures, which need to be adjusted to 100 million US dollars of R&D expenses (as in Table 3) to allow comparison between Kazakhstan and other countries.
As with the first indicator, Canada and Australia show a spike in this metric, whereas the USA maintains a stable trend.In universities and research institutes in Kazakhstan, there is still no established system for disclosing information about obtained scientific results.As noted above, since 2012, NCSTI has started collecting data on scientific and scientifictechnical activities (SSTA) results obtained within grant and program-targeted funding projects.However, these data are "raw," improperly formatted, and not available or published in open reports by NCSTI.It is important to disclose this information specifically in commercialization offices, as they are the ones that should further evaluate, patent, and commercialize it.Additionally, it is crucial for commercialization offices to transfer information on disclosed intellectual property objects to NCSTI.
3) Licensing Activity The metric "Licensing Activity," as defined in the ATTP report, quantitatively assesses the frequency or scale of transactions involving licenses, options, and transfers of rights related to disclosed intellectual property.Tab. 4 details licensing activity, specifically indicating the number of executed licensing and option agreements, as well as rights transfer agreements (LOA) per 100 million US dollars in R&D expenditures.Higher values of this metric indicate an increased level of business activity in intellectual property.Throughout the entire period, Canada led this indicator.However, since 2017, Australia has surpassed Canada, becoming the leader among the considered economies.Regarding Kazakhstan, information on licensing activities, specifically the number of executed licensing agreements, options, and transfers of rights (LOA), is not fully available.Data from the Bureau of National Statistics include information on licensing agreements and agreements on the transfer of intellectual property rights (IP) in Kazakhstan.These licensing agreements cover a wide range of IP categories, including "inventions," "utility models," "industrial designs," "breeding achievements," and "trademarks," as well as agreements on the transfer of IP rights (Tab.5).Observing the data, some fluctuations in the number of agreements over the years can be noted.For instance, fluctuations are observed from 2005 to 2009.Since 2009, the number has generally maintained an upward trend, with some fluctuations in certain years.The data indicate a variable level of activity in intellectual property transactions, with potential implications for innovation, economic development, and intellectual property management in Kazakhstan.Moreover, the significant increase in the number of agreements since 2009 may indicate growing attention to intellectual property management and commercialization efforts in the country.

4) Revenue
The "Revenue" metric evaluates the financial return obtained from transactions involving disclosed intellectual property, such as licenses, options, and transfers of rights (LOA).This metric shows the share of revenue obtained relative to research expenditures.A higher ratio indicates a more significant financial return from the commercialization of intellectual property assets.Tab.6 presents the indicator, measured as the ratio of income from LOA to R&D expenditures (%).Higher values of this metric indicate higher efficiency of investments in research into specific financial benefits from intellectual property transactions.Despite lagging in the number of commercialization transactions (Table 6), the USA's positions in terms of income obtained from these transactions are very favorableexceeding them by 2-3 times.Unfortunately, detailed data regarding the revenue metric, particularly the financial returns from intellectual property transactions in Kazakhstan, are currently unavailable.Encouraging entrepreneurs to record these data is extremely challenging under current conditions.A possible solution to this situation could be tax report sources, as tax incentives and deductions might encourage companies to make this information available.5) Efficiency Finally, the sought metric "Efficiency" reflects the degree of knowledge exchange in the system, determined by the ratio of transactions to disclosed intellectual property.A higher degree indicates improved efficiency in disseminating and utilizing intellectual property assets to promote innovation and commercialization.Tab.7 presents the Efficiency metric, measured as the ratio of LOA to intellectual property disclosures in Australia, Canada, and the United States.Australia 0,40 0,49 0,47 0,46 0,35 0,34 0,31 0,29 0,35 0,44 0,61 0,45 0,59 0,85 0,81 0,85 Canada -0,40 0,30 0,37 0,35 0,22 0,31 0,32 0,31 0,22 0,30 0,39 0,55 0,38 0,44 0,42 USA --0,28 0,26 0,26 -0,26 0,26 -0,26 0,28 0,27 0,27 0,29 0,31 0,30 Kazakhstan 20.53 18.76 1.20 2.54 8.50 1.23 12.48 2.07 It should be noted, however, that assessing efficiency for Kazakhstan, measured as the ratio of transactions to intellectual property disclosures, is impossible due to the lack of data on intellectual property disclosures.Consequently, assessing the efficiency of knowledge transfer in the Kazakhstani context is currently unattainable.

6) Startup Creation Activity
This metric quantitatively assesses the number of new companies created as a result of knowledge exchange, technology transfer, or the commercialization of research activities.This assessment metric provides insight into entrepreneurial outcomes stemming from intellectual property transactions and research efforts.Table 9 presents the startup creation activity metric, measured as the number of startups created related to intellectual property per 100 million US dollars of research expenses, covering Australia, Canada, and the United States.Higher values of this metric indicate a greater propensity for innovative entrepreneurship and economic development in the respective regions.

Conclusion
In conclusion, the cross-country analysis of commercialization effectiveness reveals diverse approaches and methods for assessing the effectiveness of technology commercialization.From financial metrics to social and environmental impact, these indicators play a crucial role in understanding commercialization outcomes.Previous studies, such as those by Kumar et al. (2015) and Sung (2009), highlight the importance of effective technology transfer and the challenges associated with it.Additionally, the works of Gibson (2014) and Sutopo et al. (2019) emphasize the broader implications of successful commercialization on economic and social development.
Despite the diversity of methods, commercialization metrics assessment in Kazakhstan faces several challenges due to limited data availability and the lack of comprehensive data sets.The absence of data on office resources, intellectual property disclosures, and revenue metrics in Kazakhstan highlights the difficulties in assessing research commercialization in the country.This aligns with the findings of Alibekova (2018) and Suleimenova (2018), who identified similar issues in their studies on Kazakhstan's commercialization landscape.Limited data availability hinders a complete understanding of the commercialization ecosystem, making informed decision-making and strategic planning difficult.Moreover, the absence of data on startup creation activity further complicates the assessment of entrepreneurial outcomes in Kazakhstan.
Comparative studies show differences in organizations dedicated to supporting and transferring scientific research across countries.For instance, in the United States, the Association of University Technology Managers (AUTM) plays a key role.In Canada, significant organizations include the National Sciences and Engineering Research Council (NSERC) and the Centers of Excellence for Commercialization of Research (CECR).In Australia, Commercialization Australasia (KCA) operates, and in South Korea, the Korea Advanced Institute of Science & Technology (KAIST) is prominent.In Europe, the leading knowledge transfer organization is the Association of Knowledge Transfer Professionals (ASTP), which aims to share best practices and develop competencies in knowledge transfer activities.
In Kazakhstan, the absence of a specialized organization providing commercialization effectiveness assessment underscores the complexities in comparing Kazakhstan's commercialization efforts with international standards.This fact highlights the need to create such a structure responsible for analyzing and assessing the effectiveness of the technology commercialization process in the country.Studies on commercialization effectiveness in different countries emphasize the importance of joint efforts by government agencies, research institutions, and industry representatives to improve data collection and reporting mechanisms.For further development, it is necessary to enhance data collection and reporting mechanisms to ensure comprehensive assessment of commercialization metrics in Kazakhstan.
Thus, to achieve success in technology commercialization in Kazakhstan, it is necessary not only to improve data collection and availability but also to develop institutional mechanisms, including the creation of a specialized organization that would evaluate the effectiveness of technology commercialization and support innovation activity in the country.The cross-country comparative analysis highlights differences in commercialization assessment patterns and supporting organizations across countries, with Kazakhstan lacking a specialized organization for evaluating commercialization results.While there are initiatives such as the Alliance of Technology Commercialization Professionals (ATCP), their effectiveness seems undermined by limited support and resources.
To address these issues and effectively assess commercialization effectiveness in Kazakhstan, alternative research methods such as data envelopment analysis (DEA) models or regression analysis may be required.These approaches can utilize available data and provide insights into the efficiency of commercialization efforts, even in the absence of comprehensive data sets.
Government expenditures on research and development (R&D); -Number of personnel involved in research and development; -Share of innovative products (goods and services) in the gross domestic product (GDP); -Volume of issued protection documents; -Number of new technologies and technical objects created and utilized; -Import and export of R&D services; -Payments for the use of intellectual property not classified under other categories; -Number of licensing agreements and contracts related to the transfer of industrial property.

Table 1 -
The bodies responsible for evaluation of technology commercialization

Table 2 -
Number of equivalent full-time rates for commercialization per institution (units)

Table 3 -
Disclosure of inventions for 100 million US dollars in R&D expenses

Table 4 -
The number of executed licenses, options, and transfers of rights (LOA) per 100 million USD R&D

Table 5 -
Number of license agreements and agreements on the transfer of rights to intellectual property (IP)

Table 6 -
Ratio of income from LOA to research expenses (%)

Table 7 -
The relationship between the efficiency of knowledge transfer -LOA and the disclosure of intellectual property

Table 8 -
Number of created startups related to intellectual property, per 100 million US dollars of research expenses (units)