Integrating MegaLinter to Automate Linting Across Multiple Codebases. A Technical Description.
Apr 9, 2025
•
Thorsten Foltz
Automate code quality with MegaLinter, SQLFluff, and custom checks in Azure DevOps CI. Supports multi-language linting and dbt integration.
If you’re not familiar with linters, or specifically with MegaLinter, please take a look at my previous article on the topic. In contrast to the previous article, this one focuses on implementing several linters using MegaLinter for Python, Docker, SQL, YAML, Bash, JSON, Markdown, Make, Terraform, the repository itself, and spellchecking. Additionally, an approach is demonstrated to implement SQLFluff not only for SQL but also for dbt. A shell script is also introduced, which tests for specific patterns in Git branch names. Finally, all of these are integrated into a pipeline on Azure DevOps for use within a CI/CD process. The following linters will be used:
bash-exec: Checks if shell files are executable. In MegaLinter, this is referred to as
BASH_EXEC.shellcheck: Provides warnings and suggestions for bash/sh scripts. In MegaLinter, this is referred to as
BASH_SHELLCHECK.jsonlint: A JSON/CJSON/JSON5 parser, validator and pretty-printer. In MegaLinter, this is referred to as
JSON_JSONLINT.jsonprettier: Code formatter, enforces a consistent style. In Megalinter, this is referred to as
JSON_PRETTIER.jsonv8r: Check the validity of Json files, if they have a matching schema defined on schemastore.org. In Megalinter, this is referred to as
JSON_V8R.checkmake: Linter and analyser for Makefiles. I like it pretty much, but it’s currently disabled, due to security issues. Last update almost two years ago. In MegaLinter, this is referred to as
MAKEFILE_CHECKMAKE.markdownlint: Checks for errors in Markdown files, and can also auto-fix some of them. In MegaLinter, this is referred to as
MARKDOWN_MARKDOWNLINT.markdown-link-check: Checks all of the hyperlinks in a markdown text to determine if they are alive or dead. In MegaLinter, this is referred to as
MARKDOWN_MARKDOWN_LINK_CHECK.markdown-table-formatter: Check markdown tables formatting and apply fixes. In MegaLinter, this is referred to as
MARKDOWN_MARKDOWN_TABLE_FORMATTER.black: A python code formatter. In MegaLinter, this is referred to as
PYTHON_BLACK.bandit: A tool designed to find common security issues in Python code. In MegaLinter, this is referred to as
PYTHON_BANDIT.flake8: A python tool that glues together pycodestyle, pyflakes, mccabe, and third-party plugins to check the style and quality of some python code. In MegaLinter, this is referred to as
PYTHON_FLAKE8.isort: A python utility/library to sort imports alphabetically and automatically separate into sections and by type. In MegaLinter, this is referred to as
PYTHON_ISORT.checkov: Prevent cloud misconfigurations and find vulnerabilities during build-time in infrastructure as code, container images and open source packages. In MegaLinter, this referred to as
REPOSITORY_CHECKOV.gitleaks: A tool for detecting secrets like passwords, API keys, and tokens in git repos, files, and whatever else you wanna throw at it via
stdin. In MegaLinter, this is referred to asREPOSITORY_GITLEAKS.kics: Find security vulnerabilities, compliance issues, and infrastructure misconfigurations early in the development cycle of your infrastructure-as-code. In MegaLinter, this is referred to as
REPOSITORY_KICS.ls-lint: A fast directory and filename linter. In MegaLinter, this is referred to as
REPOSITORY_LS_LINT.secretlint: Linting tool to prevent committing credential. In MegaLinter, this is referred to as
REPOSITORY_SECRETLINT.semgrep: Lightweight static analysis for many languages. Find bug variants with patterns that look like source code. In MegaLinter, this is referred to as
REPOSITORY_SEMGREP.trivy: Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more. In MegaLinter, this is referred to as
REPOSITORY_TRIVY.trufflehog: Find, verify, and analyze leaked credentials. In MegaLinter, this is referred to as
REPOSITORY_TRUFFLEHOG.lychee: Fast, async, stream-based link checker written in Rust. Finds broken URLs and mail addresses inside Markdown, HTML, reStructuredText, websites and more. In MegaLinter, this is referred to as
SPELL_LYCHEE.proselint: A linter for English prose. In MegaLinter, this is referred to as
SPELL_PROSELINT.sqlfluff: A modular SQL linter and auto-formatter with support for multiple dialects and templated code. Used also for dbt. In MegaLinter, this is referred to as
SQL_SQLFLUFF.terragrunt: A flexible orchestration tool that allows Infrastructure as Code written in OpenTofu/Terraform to scale. In MegaLinter, this is referred to as
TERRAFORM_TERRAGRUNT.tflint: A pluggable Terraform Linter. In MegaLinter, this is referred to as
TERRAFORM_TFLINT.yamlprettier: Look for the jsonprettier above. Same tool, just for yaml. In MegaLinter, this is referred to as
YAML_PRETTIER.yamlv8r: Look for jsonv8r. Same tool, just for yaml. In MegaLinter, this is referred to as
YAML_V8R.
Set a MegaLinter configuration file
In the root of your repository, create a file named .mega-linter.yml. This file contains the configuration for MegaLinter itself and for many, but not all, of the linters. My file:
Allow me to explain my settings shortly:
APPLY_FIXES: MegaLinter can automatically fix your code as defined in the linter rules. I don’t want that automatically, thus I decided to use “none”.
BASH_SHELLCHECK_ARGUMENTS: With “-e” a rule is excluded. In this case rule “SC2162”.
BASH_EXEC_FILTER_REGEX_EXCLUDE: Any files in the “dbt_packages” directory will be ignored by the
bash_execlinter. Ignoring this directory is necessary because it contains other Git repositories with files that might violate certain rules, often too many to fix comprehensively.BASH_SHELLCHECK_FILTER_REGEX_EXCLUDE: “dbt_packages”. Same as described above.
CLEAR_REPORT_FOLDER: Flag to clear files from report folder (usually megalinter-reports) before starting the linting process. I don’t need to keep these reports, so I set it to
true.DISABLE_ERRORS: Flag to have the linter complete with exit code 0 even if errors were detected. By default, it is set to
false, but I explicitly define it. After all, what’s the point of a linter if you disable errors?ENABLE_LINTERS: A list of all linters used in this repository. For an explanation, see above. Every linter you want to use must be specified here. Instead of enabling specific linters, you can also use
DISABLE_LINTERSto enable all linters except those explicitly mentioned.FAIL_IF_UPDATED_SOURCES: If set to
true, MegaLinter fails if a linter or formatter has autofixed sources, even if there are no errors.FILEIO_REPORTER: Upload MegaLinter reports to file.io if set to
trueFILTER_REGEX_EXCLUDE: Regular expression defining which files will be excluded from linting. Probably better to set exclusions for specific linters.
FLAVOR_SUGGESTIONS: The main drawback of MegaLinter is its heavy resource usage. While caching minimizes the impact locally, in a CI/CD pipeline, execution can take significant time. If you’re charged per execution time rather than a flat fee, this can add to your costs — though likely minimal compared to other expenses. However, not all linters are necessary for every project. To address this, MegaLinter offers a recommended configuration, which you can enable by setting it to
true.FORMATTERS_DISABLE_ERRORS: Some linters may display warnings or errors if a file is improperly formatted. I recommend always setting this to
false.IGNORE_GITIGNORED_FILES: The recommended and default setting is
true, meaning anything ignored by Git due to the.gitignorefile is also ignored by MegaLinter.LINTER_RULES_PATH: Some linters require configuration files to be stored at the root of the repository. However, many allow using a custom directory. Create a directory of your choice and specify it here.
LOG_LEVEL: How much output the script will generate to the console. One of
INFO,DEBUG,WARNINGorERROR.MARKDOWN_DEFAULT_STYLE: Markdown default style to check/apply.
markdownlint,remark-lint.MARKDOWN_MARKDOWN_LINK_CHECK_FILTER_REGEX_EXCLUDE: Linter Markdown_link_check excludes anything in directory “dbt_packages”.
In the following, any linter that simply searches for its configuration file in the .linter directory will not be described. The same applies to cases where only a specific directory is ignored from linting.
PARALLEL: Process linters in parallel to improve overall MegaLinter performance. If true, linters of same language or formats are grouped in the same parallel process to avoid lock issues if fixing the same files.
PRINT_ALPACA: Enable printing alpaca image to console.
REPOSITORY_SEMGREP_RULESETS: List of rules which are included. Find all available rules here.
SHOW_ELAPSED_TIME: Displays elapsed time in reports.
SHOW_SKIPPED_LINTERS: Displays all disabled linters mega-linter could have run. Please note, from time to time MegaLinter disable certain linters, due to bugs.
Set Linter Configuration
Some linters can be configured within the .mega-linter.yml file, while others require separate configuration files, which can often be placed in a custom directory—for example, .linters in my case. However, some linters strictly require their configuration files to be at the root of the repository. Let's start with those. Note that these are not configuration files, but ignore files: .semigrepignore, .sqlfluffignore, and .trivyignore.
My configuration files within directory .linters are:
.bandit.yml which exclude certain directories and skips rule B101.
.checkov.yml which just ignores directory dbt_packages
.flake8 the famous python linter, ignoring rules E501 and F821 as well as ignoring two files.
kics.config is set to check Docker and Terraform files, with the LTS Spark version check ignored.
.ls-lint.yml set for certain file formats rules. In my case file names must always be in lower cases or snake cases. Some directories are excluded.
lychee.toml accept response code 200 and 429. Moreover, ignore certain urls, files and pathes from check.
.markdown-link-check.json just ignore certain patterns and status codes.
.markdownlint.json enables or disables certain rules.
.proselintrc disable or enable rules.
.sqlfluff the famous sql linter. Pay attention, this is only for sql files outside of dbt. Anything regarding dbt is ignored, because dbt needs the same files, but slightly different. I will show it later. Configure sql code as you like. For example, use always leading comma or every table, view and column must be in lower case. Important: define your database on the top. In my case it’s databricks.
.tflint.hcl in this case only set a required version.
.yamllint.yml Also just a few settings. Especially the allowed length of a row.
Linters outside of MegaLinter
MegaLinter cannot incorporate all existing linters. However, they offer a solution for extending it with a custom linter of your choice (though I haven’t tested it — maybe in the future).
I created my own Dockerfile and a simple shell script to use instead. The first leverages sqlfluff for dbt, while the second ensures Git branch names follow specific rules.
Let’s start with sqlfluff for dbt. The configuration file, .sqlfluff, remains the same as before, with a slight adjustment at the beginning:
This file must now be stored in the root of your dbt project, rather than the root of the repository. Additionally, if you need to exclude certain directories or files from linting, place a .sqlfluffignore file in the same location.
The needed docker file:
Additionally, the project file pyproject.toml is required, along with entrypoint.sh. The latter is stored in the docker subdirectory alongside the Dockerfile, while the project file is placed in the root of the repository.
The project file:
The shell script entrypoint:
Please note that sqlfluff will establish a connection to your database, requiring a functional dbt profile. If you run the linter locally, you can use your default profile. However, for a CI/CD process, you must specify a profile—such as profiles-pipeline.yml—configured with environment variables.
Locally the docker can be executed:
To run automated fixes, simply extend the command with fix.
In a pipeline, make sure to update your home path to match the location of the profile, which must be stored in your repository.
Another linter checks Git branch names. Personally, I prefer branches to follow a specific pattern — starting with a prefix like feature, fix, or test, followed by a slash and then a descriptive purpose, ideally including a ticket number.
For example: feature/db-123-financial-data.
My file check_git_branch_name.sh stored within the directory .linters :
Execution
I’ve already explained how to execute sqlfluff with dbt using Docker, and running a shell script should be straightforward.
MegaLinter requires npx and can be executed with:
I recommend consolidating all these commands into a single Makefile or shell script. You can also extend it by adding the fix option to allow linters to automatically resolve any issues they can.
For example:
In an Azure DevOps pipeline, it could look like this:
Final remarks:
MegaLinter Updates: If you don’t set a fixed version for MegaLinter, you will receive updates automatically. While this can bring improvements, it might also introduce new required fixes that weren’t necessary before, potentially causing inconsistencies in your workflow.
Extensive Linter Coverage: MegaLinter includes a wide range of linters for various programming languages. Be sure to explore it and take advantage of additional linters for other types of files in your project. It can help ensure code quality across different languages and file types.
Docker Image Size: MegaLinter is quite large, and pulling the full image can be around 10 GB. While this isn’t a problem locally, as it only needs to be pulled once and Docker caching is used, in a CI/CD pipeline, it would fetch everything every time. Consider using a specific linter flavor to optimize the pipeline build time and resource usage.
Linting in a Team: Linters are valuable when working in teams, as they enforce consistent coding practices and improve overall code quality. However, they can also lead to many discussions about specific rules. Be cautious when disabling checks — some developers might be inclined to ignore formatting rules or fix issues only when necessary. It’s important to maintain discipline in adhering to standards. If you start making too many exceptions, you undermine the value of the linter itself.
By keeping these considerations in mind, you’ll have a more efficient and consistent linting process, both locally and in your CI/CD pipeline, while also fostering a culture of quality coding within your team.
Latest
Portable by design: Rethinking data platforms in the age of digital sovereignty
Build a portable, EU-compliant data platform and avoid vendor lock-in—discover our cloud-neutral stack in this deep-dive blog.
Cloud Independence: Testing a European Cloud Provider Against the Giants
Can a European cloud provider like Ionos replace AWS or Azure? We test it—and find surprising advantages in cost, control, and independence.
Stop loading bad quality data
Ingesting all data without quality checks leads to recurring issues. Prioritize data quality upfront to prevent downstream problems.