Skip to content
#Architecture

Large-Scale Automated Refactoring Using ClangMR (Category Architecture)

#Architecture #Leadership #Software #SoftwareDevelopment #AI #Engineering

Lately, I’ve been learning a lot about how Gen AI helps in SDLC. But sometimes we need to go back and understand how we lived before generative intelligence and what we did to improve refactoring. And if you dig into it, you can see that these old and proven approaches are actively used in modern times to provide predictable results of a combination of stochastic and deterministic approaches. In fact, this is how I came to read. whitepaper 2013 year Google guys. Here they talked about large-scale refactoring of their code base in C++, for example, from an older version of the standard to a newer one. Interestingly, their codebase scale required that they immediately consider the possibility of parallelizing the work, for which they used Map Reduce. (This is what MR stands for at the end of ClangMR.). But now it’s time to move on to the main ideas from this article.

The fact is that the ClangMR framework solves the problem of technical debt in large C++ codebases through semantically secure refactoring, which is well paralleled. The key points of the approach are

  • Use of AST (abstract syntax tree)abstract syntactic trees. Unlike regex tools, this allows you to describe changes much more accurately, for example, you can easily distinguish methods with the same names in different classes.
  • Scalability. Distributed processing via MapReduce reduces refactoring time from months to hours.
  • Repeatability. Support for incremental changes when authors use iteratively to refactor tens of thousands of files in pieces

ClangMR implementation is based on three whales - Indexing (indexation): compilation of codebase in AST with storage in distributed database (e.g., Google's Bigtable).

  • Node matching (juxtaposition): Developers define AST templates (e.g. memberExpr(hasName("OldMethod")) And callbacks for edit generation. Interestingly, this part consists of a description of the pattern for matching parts of the tree, as well as a callback for applying changes. (Callback may decide that no changes are required.)
  • **Sorce code refactorer (modification)**Local application of edits with conflict resolution and autoformatting through ClangFormat.

This article and the approach described in it led to several consequences. n

  • Development of Clang/LLVMComponents such as the AST matcher API became the basis for Clang-Tidy and Clang refactoring engine
  • Changing refactoring practices: Popularization of AST approaches that influenced tools like Coccinelle for the Linux kernel. The evolution of the standard library: The accelerated adoption of modern capabilities has put pressure on compiler vendors.

Overall, the article was very interesting for 2013 years. But if you talk about the mix of this approach with Gen AI, you can see the report of the guys from Uber. 2024 yearThis Year in Uber’s AI-Driven Developer Productivity Revolution", in which they talked about, among other things, the migration from Java to Kotlin, where the AST matcher mix was actively used, as well as Gen AI, which helped generate the rules of transformation:) More can be found in my review of this report. (1 and 2)

#Architecture #Leadership #Software #SoftwareDevelopment #AI #Engineering