My Code is Clean Starting Today Using Verissimo SystemVerilog Linter
Description
We have coding guidelines. Sometimes we read them, oftentimes we forget. We do code reviews. Sometimes carefully, oftentimes deadlines rush us to postpone improvements. Sometimes we spend time tracking issues, oftentimes no time for fixing. Sometimes we use code analysis tools to automate checking and tracking, oftentimes we are overwhelmed with the results. We cannot cheat the tool, it is in our face: hundreds, thousands, even tens of thousands of loose ends in our code. We are paralyzed. How do we move forward? This presentation provides recipes to cope with this nightmare and keep your code clean. Starting today.
Explore the design and verification tools: https://www.dvteclipse.com
Or request a license: https://www.dvteclipse.com/request-license
Transcript
Hi there, I'm Alex Marin, I'm an R&D Team Lead at Amiq EDA and this presentation is about getting results fast using the Verissimo linter.
Overview of Topics
Let's start with a quick overview of the topics that I'm going to cover in this presentation.
First, a little background on Verissimo and a short overview of the tool.
Then, we'll state the problem, feeling overwhelmed by the first run of Verissimo. You may get thousands or even tens of thousands of failures in your project. This is quite discouraging and the question is how can you manage that huge fixing effort?
We'll move on to the solution that we propose, focus on keeping the new code clean. And this is done by what we call Delta Analysis or Delta Linting.
Then, we'll look into another technique, a timeline-based analysis or in short, recent failures analysis.
Last but not least, we'll talk about how you can boost the fixing of lint failures using the autocorrect capabilities of Verissimo.
And moving forward, we'll take a look at some real numbers from case studies of Verissimo runs on some real-life projects.
And at the end, we'll have a quick overview of the Verissimo best deployment practice that is in a continuous integration flow.
Overview of Verissimo SystemVerilog Linter
Now let's have a quick overview of the Verissimo SystemVerilog Linter. It helps you enforce methodology and make sure that coding guidelines and best practices are followed, thus improving the project quality and maintainability. Think of all those debug hours you spend on some problem, only to find out that the root cause was something you should have detected with a careful code review. Verissimo automatically catches such issues upfront.
As this presentation is recorded, Verissimo has almost 700 built-in rules derived from real-world customer requests, so a lot of know-how and real-world experience is packed in the tool and new rules are being added every week. A lot of the rules have parameters that you can tweak and adjust.
Particularly relevant for verification engineers is the set of dedicated UVM rules that help you stick to the best practices of the methodology.
The tool runs both in batch mode and GUI mode, integrating with the DVT Eclipse IDE. Verissimo also provides powerful reporting, analysis and fixing mechanisms such as the HTML report or custom-made reports, delta linting, timeline analysis and revision control systems integration, and even automatic correction of failures.
Getting Started with Verissimo
So, how about getting started with Verissimo? Well, that's really easy. Just pick the rules you want to enforce from Verissimo's rule pool or simply use one of the predefined built-in rulesets and pass your usual simulator arguments on the Verissimo command line. That's it.
You can start fixing the failures by looking at the generated HTML report. Here you can see an example of the generated HTML report dashboard. On the left side you have a filters panel, then a table with all the rules and their status and an executive summary on the right side where you can see the total number of checks, how many have passed or failed, the total number of failures and total fixing effort.
Below you have some tables with top statistics, such as top failed checks and top failed files. Then you can drill down by filtering, sorting, searching, and so on.
For example, you can filter the failures by a specific rule or by some specific file in which they were reported. You can also filter by the revision control author, for example, if you want to see all the failures on code last modified by yourself or by some other engineer in your team. Under the hood, it's actually correlating the failures with information from the revision control system, such as git blame.
All these analysis techniques help you focus on a specific set of failures, but still, when you actually want to start fixing, the numbers can be a real bummer, especially when your summary shows something like this: 20,000 failures, 2,500 hours, that's about 100 days or about half a year of work.
The Problem: Why Do We Get So Many Failures?
So let's talk a bit about why do we get so many failures. Actually, the answer is quite simple, because you almost never start from scratch. Most of the time, you reuse entire components, either across projects or project revisions and this is generally called legacy code. Or you use third-party IPs that you either get from another internal team or from an external vendor.
Legacy code is largely irrelevant, or at least it's regarded as low priority versus the actively developed components, and it generally follows the well-known rule, if it works, don't break it, especially since it's been proven to work in previous projects. Also, many times ownership for legacy code is lost or diluted, so engineers tend to regard failures in such code as not my problem right now.
Also, third-party IPs are totally out of your control and failures are completely irrelevant, unless you ask the VIP provider to also enforce coding guidelines and have an automated linting QA process in their release pipe.
Anyhow, back to our problem. We have 20,000 lint failures to deal with. That's huge. It looks scary and unmanageable, right? Where should I start? How can I make this effort tractable?
If you think about it this is also a problem of perception. Engineers work hard to meet tight deadlines, so having a tool that tells you you've got even more work to do can be quite depressing and frustrating.
Solutions: Waivers and Delta Linting
Well, waivers are one thing that comes to mind. At least for legacy code and third-party IPs we could selectively ignore failures or simply ignore everything else except our code, the actively developed parts of the project. Or even better, we could create some pre-waivers that get applied upfront and entirely skip the analysis of some parts of the code, therefore leading even to some lint-time improvements.
However, all this takes some time to set up and maintain and we'd probably still be left with a significant amount of failures. So how about getting something done today? And this is the solution, delta analysis of the linting results, or in short, delta linting.
Delta Analysis of Linting Results (Delta Linting)
Basically, you compare the linting results on the current version of your code with a baseline. This way, you can see how the linting evolved, more or less like when you look at how a source code file evolved in the history of a revision control system. It's a comparison of the linting report before your changes, and that is the baseline report and after your changes, which is the current report.
The baseline could be time-based, say each Thursday you take some time to review the new failures that crept into your code during the week's time of development. Or it could be revision control-based. For example, run the linting before every commit and compare it with the previous one, which in this case would be the baseline. Or you could do it only when merging back into the master branch and always have the master set as baseline.
What you achieve with this type of analysis is that you can now focus on the new code and fix just the new failures. The number of failures drops from thousands or tens of thousands to maybe just tens of them. Suddenly, fixing becomes manageable and clearly makes more sense, especially since the failures are in code you just wrote or modified, code that is fresh in your memory right now.
You can regard this as a fast lane to the added value of Verissimo. You can immediately leverage the benefits of Verissimo on your code. And you can stop accumulating even more technical debt. You can make sure that your code is clean from now on, starting today!
How Is This Done?
So, how is this done? First, generate the baseline linting report. By the way, in a continuous integration flow where Verissimo runs automatically on every commit or check-in, you always have it at hand. We'll get back for a deeper dive into this flow a little bit later.
Then write some new code, and when you're done, generate the current linting report and the comparison report. As you can see, it's fairly straightforward. You just need a couple more switches on the Verissimo command line.
Once you have the compare HTML report generated, you can activate the new failures filter. This shows only the failures that were not present in the baseline. Basically, the issues introduced by recent changes, the stuff you want to fix now.
In this snapshot you can see that we have only 23 new failures with an estimated fixing time of 46 minutes. That's quite compelling!
You can also look at the new fixes. After all, it's not like you never accidentally spot an issue and fix it since you're there. In this particular example, 14 failures from the baseline have also been fixed in the current version of the code.
By the way, not all the failures that show up in the current report are necessarily new. Maybe you added a new rule to your ruleset. And similarly, not all the failures that disappeared are actual fixes in the code. Maybe you added a waiver.
Now, all these dimensions can easily be explored using the filters available in the comparison report.
However, most of the time, you'll focus on the new failures and on actually fixing them. But before going into the fixing part of the presentation, let's also take a look at timeline analysis.
Timeline Analysis (Recent Failures Analysis)
Let's say a lighter flavor of delta linting. Timeline analysis basically uses the time of code changes as the baseline. The nicest part about it is that it requires zero effort to set up. Well, it does take an extra switch in the command line, -include_creation_date.
What it does is to give you the ability to filter failures based on code change timestamp. Verissimo draws this information either from the revision control system—git blame, for instance can provide the timestamp for each line of code or if the revision control system can't provide such info, directly from file system timestamps.
To make the best of it, cross this information with author info, if available. Again, git blame, for instance can provide the author who last touched the line. This way, you can instantly get the answer to this question, what are the failures in the code that I touched last week? That's pretty cool, right?
However, there is a downside. This technique is less accurate than delta analysis because it will not get rid of old failures on newly touched sections of code. Imagine a failure that has been there for a year. Now you just add a printout on the same line or change the name of a variable. The old failure is on a recently changed line of code, so it will also show up in the last week filter.
In this snapshot, you can see how the last week filter quickly brings down the number of failures to just 8 out of over 300 that we initially had. Not surprisingly, these failures are aggregated around the GPIO files, which is the particular component that we actively worked on during the last week.
Boosting the Fixing of Lint Failures Using Autocorrect
Finally, it's time for fixing and Verissimo can help with that too. Fixes for some Verissimo rules can be automated, especially commonly used mechanical rules which are also the often failing, annoying types of checks. Think about code formatting, naming conventions, `ifdef guards, forgetting to call super in a virtual method override and so on.
Well, Verissimo Autocorrect turns a significant part of the failures into low-hanging fruit. When dealing with many failures, it can save even hours of tedious work. Autocorrect can be performed in batch mode or in the DVT Eclipse IDE.
In the DVT Eclipse IDE, the process is supervised. You can preview and approve the changes before they actually get applied. Here we can see a simple example. The rule says all enumerated item names must be uppercase. Just right-click on the check and pick Fix Failures. The tool does all the magic and presents you the proposed changes.
Of course, the changes include the enum declarations and their usages as well. A list of all the affected files is shown, and for each file you can get a comparison of the code before and after correction is applied.
Case Studies: Real Numbers from Verissimo Runs
Okay, so now let's look at some numbers. First, a small IP. To get an idea about the size of this project, we added some measurements.
- 300 files
- Less than 150,000 lines of code
- Containing about 400 types, that is, classes, structs, enums, modules, interfaces and so on
The ruleset is actually the built-in basic Verissimo ruleset, comprising of 59 rules. The baseline linting results show that 28 checks have failed with a total of 335 failures and an estimated fixing effort of about two and a half days.
After coding some enhancements, the current report and the comparison with the baseline were generated. Only 8 new failures with an estimated effort of 28 minutes. That's easy, right?
After cleaning up those 8 failures, we've taken another delta. And looking at the new fixes this time, we're in for a surprise. 30 additional new fixes. How come?
Well, the engineer who kindly provided this data said "I couldn't help fixing some other old low-hanging failures in the areas where those 8 new failures were located". Basically, since I'm there, and the code is fresh in my head, why not tidy up a few more things? It's an opportunity you can't miss.
On a much larger project, weighing in at:
- 2,500 files
- Almost 600,000 lines of code
- 4,000 types
We have over 20,000 failures in the baseline with an estimated effort of about 100 days or over 6 months. Of course, assuming that you're not working all weekdays 24 hours a day.
Similarly with our previous example, a baseline was generated, some changes were done, then current and comparison reports. Not surprisingly, results are similar, because the new failures are proportional to the extent of our changes and not to the absolute size of the project. As expected, clutter is out of our way and we can focus on clean new code, plus an opportunity bonus of 41 additional fixes.
Verissimo Best Deployment Practice in Continuous Integration Flow
Finally, I'd like to do a short overview of the best practice recommended for the deployment of Verissimo, a continuous integration flow. In such a flow, Verissimo runs on every commit or every time you check into your revision control system, on a regular schedule, for example nightly. It can send custom reports over email or inject linting results into a database for big data, fancy graphs type of analysis. It can even act as a revision control check-in gate, for example, with a reduced set of critical rules.
Furthermore, this gives you an always up-to-date state of the linting results and a set of baselines ready to use for delta analysis.
The next step of the flow is analysis. Analyze using all the HTML report features, sorting, filtering, searching, blame information, fixing effort estimation and techniques such as the ones we've discussed today, delta and timeline analysis.
Of course, the last step is fixing in GUI mode. Fix using autocorrect, then use the incremental flow, fix a failure, save the file, and reapply only the failed checks for instantly updated results. Maybe add waivers for the corner cases and you're done!
It's time to see it in action!
Try Verissimo Yourself
Verissimo runs continuously every 6 hours on the UVM library source code and on the Open Hardware Core 5 Verif RISC-V Processor Test Bench. Go to the Verissimo product page on dvteclipse.com or simply scan the QR codes on the screen to play with the Verissimo HTML report right now.
Conclusion
Thank you very much for your time and attention. I really hope you've enjoyed the presentation and learned something interesting about Verissimo and how to get your code clean starting today.