Git guardian что это
Git Guardian Component
This component helps querying remote services, like BitBucket and GitHub to get a list of repositories and helps cloning them locally for use or backup. It provides an API and a command line interface utility, read below for more information.
The requirements are specified in the composer.json file:
This component is available for installation through composer. From command line run:
Or add the following to your composer.json in the require section:
For more information check the project page on Packagist.
You can use the Gioffreda\Component\GitGuardian\GitGuardian class to bulk clone your repositories into a destination. The following is an example on how to do so:
The GitGuardian class emits a range of events that allows you to monitor the status:
You can use the Gioffreda\Component\GitGuardian\Adapter\BitBucketRemote class to fetch the list of repositories owned by a specified user. If you provide the OAuth2 client ID and secret you can fetch the private ones as well. Here’s an example on own to use the class:
Available events emitted by the remote adapter are:
Command Line Interface
You can use the provided command bin/git-guardian to clone your repositories or list the known ones that you’ve already cloned locally. The command line interface is built using the Symfony Console Component and can be easily integrated in your Symfony project.
Cloning all repositories
For example to clone locally your BitBucket repositories, including your private ones, run the built-in command like so:
This will clone all repositories of the given user locally. You can specify the destination directory. You can provide more than one username or team name and the below command will clone all repositories that belongs to those users or teams. This is an example:
The command above will clone not only the public repositories for those users or teams, it will clone everything you have access to. If you do not provide the client ID and secret the command will clone only public repositories:
Another example for cloning locally your GitHub repositories, including your private ones, is:
The way GitHub authenticates is slightly different, so you have to provide your personal access token and your username as first user in the list of users/organisations you want to clone. This will clone all private repositories you have access to as well as the public ones for any other users you give on the command line. Since GitHub handles users and organisations differently you have to provide them as follows:
You can clone just the public repositories without providing your personal access token, like so:
Note that now your username requires to be identified as user and not organisation, otherwise the command will throw an error.
The help looks like below:
List all known repositories
To list all known repositories you can run:
You can specify a different adapter, the destination where the repositories have been cloned and a different format. Available formats are:
You can run the unit tests with the following command (requires phpunit):
This software is distributed under the following licenses: GNU GPL v2, GNU GPL v3 and MIT License. You may choose the one that best suits your needs.
About
A set of tools for backing up all repositories from remote services like BitBucket and GitHub.
Git Guardian
Full-featured free secret scanning for open source and small teams on GitHub and GitLab
This is a hands-on introduction to Git Guardian (“GG”), a utility to detect (monitor) API keys and other credentials and secrets exposed in source code on public SaaS or private (internal/on-prem) GitHub.
PROTIP: Know the Glossary of technical terms before you dive into other docs, at:
https://docs.gitguardian.com/internal-repositories-monitoring/glossary
“Why” and FAQ questions are answered in a blog article that’s also called “Learning Center” at
https://www.gitguardian.com/secrets-detection.
All GG’s source is kept under Github account
https://github.com/GitGuardian.
GG’s Leak Mitigation Checklist and Good development practices page is useful if you just leaked sensitive information in public source code. It lists what to do for each of the most common services: Alibaba, Algolia, AWS, DigitalOcean, Google, GitHub, GitLab, Heroku, HubSpot, Mailgun, SendGrid, Slack, Twilio, Twitter
Social
At time of writing:
https://twitter.com/gitguardian?lang=en lists 814 followers
https://www.linkedin.com/company/gitguardian/ lists 36 employees:
CEO Jérémy Thomas (oo-de-lally) is based in Paris and San Francisco
Eric Fourier is co-founder.
https://twitter.com/PonicodeDev at #GitHubUniverse 2020.
Mackenzie Jackson (@advocatemack), GG’s Dev Advocate, wrote blog articles which are repeated in the website:
In my blog article here, I wrote down the steps I took on my attempt at installing and making use of the GG web GUI and tools.
Sign up and get API Key from your GG Dashboard
Click the blue “SIGN UP FOR FREE / Internal Monitoring”.
If you have a repository on GitHub account, specify it.
Signing up adds a browser cookie to your account’s dashboard at URL https://dashboard.gitguardian.com
Logic in the JavaScript would redirect your browser to your own account’s dashboard.
Get API key
Click “Crate new API key” for an API key such as (for example):
PROTIP: You don’t need to save that key if you’re only using the GUI page shown in the next section.
Alternately, if you want to run a scan anywhere else than the GUI you’re on, follow instructions below, then return here.
Other Integrations
Click the “INTEGRATIONS” icon on the left menu of your GG Dashboard.
By default, you’re installed to use GitHub.com SaaS.
Scan sample code on GUI Dashboard
Get a taste of the response output by using the Dashboard to run a scan of GG’s sample Python and JavaScript programming code containing AWS and MongoDb secrets.
Click the “+” at the top of your Chrome browser window to open another browser window to the GG’-provided sample Python program files containing AWS and MongoDb secrets at:
Click “Raw”, click on any line of the file. Press control+A to select all. Press control+C to copy what has been selected.
Click the blue “Scan”.
Click the blue icon to the right of the “API response” field to copy GG’s response (in JSON format).
Review bucket_s3.py scan result
In an editor, create empty file bucket_s3.py.json and paste the result file in it…
Notice the AWS Client Id and Client Secret in this line:
Also notice the database username (“testuser”) and password (“hub24aoeu”) in this line:
PROTIP: Pasting would remove line breaks:
CAUTION: Responses contains secrets you don’t want others to see (which is why we scanned GG’s sample code). So please don’t format to human-readable (indented) JSON using public web pages such as jsonformatter.curiousconcept.com.
Review bucket_s3.py scan result
Repeat the above procedure, except use the contents at
Notice the secret in this line:
There is also an in-line secret in this line:
Incidents
Click the “INCIDENTS” icon at the left menu on your GG Dashboard.
Note that text scanned in the GUI doesn’t count as incidents of secrets found.
Scan my own (vulnerable) repos
Click the “PERIMETER” icon at the left menu on your GG Dashboard.
Click “ADD SOURCES” if you want to specify more repositories to scan.
I specify to repos which were created to contain vulnerabilities, including secrets:
Watch and click your stopwatch when “100%” appears next to “REAL TIME MONITORING”.
NOTE: Sources are removed from the list after scanning.
Check or uncheck the type of health determined for each source to be displayed.
NOTE: Icons under “PROTECTION” identify whether each source is being monitored.
Mouse over or click the time icon to see a pop-up of the date/time of last scan.
Open email “uncovered secrets”
In your email client, open the email from “support@gitguardian.com” with subject
Click a repo listed under “repository is affected:” to open the repo on your default browser.
QUESTION: “17 secrets”.
Try Business Collaboration Pricing
PROTIP: GG scanning is free on SaaS for open source and up to 25 developer teams.
View the comparison of plans at SETTINGS > General.
PROTIP: The free plan has a limit of 1,000 (1K) calls per month.
The “BUSINESS” plan has a limit of 10,000 (10K) calls per month.
Notice the difference between FREE and “BUSINESS” GG accounts is the
“Private collaboration repositories”.
“Collaborative” repos have “Teams” on the organization’s menu:
Within a Team is this menu with Members and Repositories associated with that Team:
Start your business trial at SETTINGS > General by clicking the blue “Start 30-day trial” button.
PROTIP: Git-guardian does not ask for your credit card up-front.
A Business plan enables you to more run options:
Invoke from CLI scan of github.com repos
Highlight and copy these two lines into your Clipboard.
The API_URL stays the same.
In the editor, also open file .env in your laptop’s Home folder.
If the file doesn’t already exist, create a new file with that name.
NOTE: https://github.com/GitGuardian/gg-shield#installation mentions addition variables:
Save API to invoke from CLI scan of github.com repos
With your editor open on file .env, highlight “____FILL ME____” and replace it with your invisible Clipboard.
Alternately, if you are running on GitHub Actions, navigate to your project settings and paste the GITGUARDIAN_API_KEY secret there.
Open a Terminal window.
You can run my script that does all the following:
Install and update using pip
Install Python 3.6 and newer
In Terminal, navigate to the folder obtained from GitHub.
If that is valid, install gg-shield using pipenv
Check if using virtualenv:
The API
The APIs called are defined at https://api.gitguardian.com/docs which provides an on-line API specification and Authentication description.
Use local Git Hooks to run on Pre-commit
Now that ggshield is installed:
Install git pre-commit hooks in local (not global) mode:
Run the last build in CI, with defaults:
Run GitHub Actions
In Terminal, create/navigate to a folder holding repositories created/cloned.
Manual run steps
In Terminal, create/navigate to a folder holding repositories created/cloned.
Invoke scanning of public repo
[1] For running on a public Github.com repo, GG has a secrets scanning API library (written in Python 3.5+) at
https://github.com/GitGuardian/py-gitguardian
The README says GG scans for “200 types of secrets”.
Install and run on Github Actions
Source code for the GG scanning engine CLI code is at
https://github.com/GitGuardian/gg-shield
It is invoked either locally on your laptop or invoked from GitHub Actions Marketplace at https://github.com/marketplace/actions/gitguardian-shield-action
Create an API key on the API Section of your GG dashboard.
To configure your .gitguardian.yml file:
Click “Raw”, click on any part of the file, press control+A,
Add line(s) under paths-ignore to specify wild cards (*)
Run within on-prem GitHub Enterprise instance
GitGuardian Private Repository Monitoring is a Kubernetes application. You can install the software on an existing cluster or use our installer that has an embedded, production-ready Kubernetes distribution packaged with it. See https://docs.gitguardian.com/internal-repositories-monitoring/self_hosting/replicated_installation
Setup Single Sign On (or SSO) allows you to manage your workspace authentication and membership via a third-party identity provider.
Install GG app on your GitHub Enterprise on-prem. server.
GitGuardian integrates with GitHub Enterprise instance through a GitHub app that we need you to create. GG enables you to do so programmatically via GitHub manifest. This will ensure that your GitHub App is created with all the appropriate rights.
Provide GG with your GitHub Enterprise url.
Create a service account email and GitHub service account
WARNING: The GitHub app will be owned by the GitHub user who created it. We therefore recommend that you subsequently transfer ownership of the GitHub app to a bot user or a GitHub organization.
Once created, you will be prompted to install the GitHub app on the GitHub organization of your choice. The installation flow even allows you to individually select repositories that you would like to give GitGuardian access to!
QUESTION: Adding custom scanning rules
QUESTION: Enterprise-wide Trend analysis
Others must know: please click to share:
Secrets Detection Engine
Philosophy#
Secrets detection is probabilistic: some secrets are easier to find than others. There is a trade-off between low number of false alerts and low number of missed credentials (precision/recall trade-off)
Our secrets detection engine has been running in production since 2017, analyzing billions of commits coming from GitHub. Since day one we began to train and benchmark our algorithms against the open source code. It allowed GitGuardian to build a language agnostic secrets detection engine, integrating new secrets or new way of declaring secrets really fast while keeping a really low number of false positives. We are also collecting feedbacks from the alerts we are sending including the pro bono alerts:
We are currently implementing two types of detectors:
Specific detectors: we implement an algorithm dedicated to looking for one specific type of secret like AWS keys, Postgres URI or SMTP credentials. This type of detector aims at having high recall and precision for the targeted type of secret. The downside of this approach is that we have to implement a lot of specific detectors to have a good coverage of all existing secrets.
You can of course choose what detectors you want to allow or deny in our different web applications.
How it works#
We currently have the following ban-lists in place:
Tao of our secrets detection engine#
Main features#
Broad coverage with 200 specific detectors#
We have developed the vastest library of specific detectors being able to detect more than 300 different types of secrets (10 times more than the current competition). You can find the exhaustive list here.
Detecting secrets with multiples matches and multi-line secrets#
GitGuardian supports the detection of multi-line secrets such as private keys.
For example we are able to detect AWS keys from the following snippet and the output will be two matches: one for the client id and one for the client secret.
Example for database credentials:
Automatic de-duplication#
For example the following content slack_token=»xoxp-198947049743-7861195093-830655328819-9d40a979cac97bccf1190afb660b37e1″ triggers two detectors: our slack_user_token detector and our detector catching generic secrets (token=
Detecting prefixed Base64 encoded secrets#
Recent frameworks like Kubernetes use Base64-encoded secrets and it is becoming pretty common to see Base64-encoded secrets. We are able to detect prefixed secrets encoded in Base64 like private keys.
Add insights to secrets to ease the work of security engineers and analysts#
We attach insights to secrets so the application security analysts can have more context information for the remediation. Currently we support the following insights:
Limitations#
Architecture#
Our secret detection engine is based on the following logic:
We then use PostValidators to validate if our secrets candidate are real secrets. We configure PostValidators for each Detector so we can achieve the best trade-off between recall and precision. You can find more information about all the PostValidators we currently use and support here.
Our Scanner is a collection of detectors that scans a Document and yield Secret objects.
Git guardian что это
GitGuardian Shield: protect your secrets with GitGuardian
GitGuardian shield (ggshield) is a CLI application that runs in your local environment or in a CI environment to help you detect more than 300 types of secrets, as well as other potential security vulnerabilities or policy breaks.
GitGuardian shield uses our public API through py-gitguardian to scan and detect potential secrets on files and other text content.
Only metadata such as call time, request size and scan mode is stored from scans using GitGuardian shield, therefore secrets and policy breaks incidents will not be displayed on your dashboard and your files and secrets won’t be stored.
You’ll need an API Key from GitGuardian to use ggshield.
Add the API Key to your environment variables:
Currently supported integrations
Install and update using pip :
ggshield supports Python 3.6 and newer.
The package should run on MacOS, Linux and Windows.
You’ll need an API Key from the GitGuardian dashboard to use ggshield.
Add the API Key to your environment variables:
ggshield scan is the main command for ggshield, it has a few config options that can be used to override output behaviour.
ggshield scan has different subcommands for each type of scan:
CI : scan each commit since the last build in your CI.
No options or arguments
Commit Range : scan each commit in the given commit range
Path : scan files or directories with the recursive option.
Pre-commit : scan every changes that have been staged in a git repository.
ggshield scan pre-commit
No options or arguments
Repo : scan all commits in a git repository.
Docker : scan a Docker image after exporting its filesystem and manifest with the docker save command.
The install command allows you to use ggshield as a pre-commit or pre-push hook on your machine, either locally or globally for all repositories.
You will find further details in the pre-commit/pre-push part of this documentation.
Warning: Using this command will discard any comment present in the config file.
Show remaining quota of the workspace.
API Status command
Show API status and version.
Configuration in ggshield follows a global>local>CLI configuration scheme.
Meaning options on local overwrite or extend global and options on CLI overwrite or extend local.
ggshield will search for a global config file in the user’s home directory (example:
/.gitguardian.yml on Linux and %USERPROFILE%\.gitguardian on Windows).
Old configuration of matches-ignore with list of secrets is deprecated but still supported :
Some configurations on ggshield can be done through environment variables.
Environment variables will override settings set on your config file but will be overridden by command line options.
At startup, ggshield will attempt to load environment variables from different environment files in the following order:
Only one file will be loaded of the three.
Reference of current Environment Variables that affect ggshield :
GitGuardian shield can be configured to run on your on-premises dashboard, request an API key from your dashboard administrator.
You can modify your environment variables to include:
Useful for ignoring a revoked test credential or a false positive, there are three ways to ignore a secret with ggshield:
⚠ this will also ignore the secret in the GitGuardian dashboard.
Secrets can be ignored in code by suffixing the line with a ggignore comment.
⚠ Your secret will still show up on the GitGuardian dashboard as potential incident.
You can use the ignore command to ignore the last found secrets in your scan or directly add the ignore SHA that accompanies the incident or one of the secret matches to the configuration file
⚠ A secret ignored on the GitGuardian dashboard will still show as a potential incident on ggshield.
Ignoring a detector
⚠ Your secret will still show up on the GitGuardian dashboard as potential incident.
The pre-commit framework
In order to use ggshield with the pre-commit framework, you need to do the following steps.
Make sure you have pre-commit installed:
Then install the hook with the command:
Now you’re good to go!
Another way is to add SKIP=hook_id before the command:
The global and local pre-commit hook
To install pre-commit globally (for all current and future repos), you just need to execute the following command:
It will do the following:
You can also install the hook locally on desired repositories. You just need to go in the repository and execute:
If a pre-commit executable file already exists, it will not be overridden.
If you already have a pre-commit executable file and you want to use ggshield, all you need to do is to add this line in the file:
If you want to try pre-commit scanning through the docker image:
Do not forget to add your GitGuardian API Key to the GITGUARDIAN_API_KEY environment variable of your project or development environment.
⚠ Pre-push hooks will not scan more than 50 commits to avoid developer interruption by default.
Pre-push hooks are executed just before git push sends data to the remote host. It will pickup and scan the range of commits between the local ref and the origin ref.
If incidents are detected in this range the push will be cancelled.
With the pre-commit framework
In order to use ggshield with the pre-commit framework, you need to do the following steps.
Make sure you have pre-commit installed:
Then install the hook with the command:
With the install command
To install the pre-push hook globally (for all current and future repos), you just need to execute the following command:
It will do the following:
You can also install the hook locally on desired repositories. You just need to go in the repository and execute:
If a pre-commit executable file already exists, it will not be overridden.
Now you’re good to go!
A pre-receive hook allows you to reject commits from being pushed to a git repository if they do not validate every check. Refer to our learning center for more information.
You can find ggshield‘s pre-receive hook samples in the doc/pre-receive.sample and doc/pre-receive-docker.sample.
ggshield‘s pre-receive hook can be skipped if the developer passes the option breakglass to the git push.
For this setting to work the remote must have push options enabled. ( git config receive.advertisePushOptions true ).
Install ggshield git pre-receive hook
This pre-receive hook requires the host machine to have python>=3.8 and pip installed
Install ggshield from pip: pip install ggshield
How do I add ignored matches and use a custom config in this pre-receive hook?
Install ggshield git pre-receive hook with docker
For the pre-receive hook to work, the directory where the repositories are stored must also be mounted on the container.
The GitGuardian Shield docker scanning tool ( ggshield scan docker ) is used to scan local docker images for secrets present in the image’s creation process ( dockerfile and build arguments) and in the image’s layers’ filesystem.
You may be interested in using GitGuardian’s GitLab integration to ensure full coverage of your GitLab projects as well as full git history scans and reporting.
Configuring GitLab pipelines to use ggshield is as simple as adding a step to your project’s pipeline:
Do not forget to add your GitGuardian API Key to the GITGUARDIAN_API_KEY environment variable in your project settings.
For ggshield to scan every commit in a merge request pipeline the CI must clone the full repository instead of just fetching the branch. The following snippet ensures this behavior.
You may be interested in using GitGuardian’s GitHub integration to ensure full coverage of your GitHub projects as well as full git history scans and reporting.
ggshield’s support of GitHub comes in the form of GitHub actions.
The action for this repository is hosted at ggshield-action.
Configuring a GitHub workflow to use ggshield is as simple as adding a step to your project’s workflow:
Do not forget to add your GitGuardian API Key to the GITGUARDIAN_API_KEY secret in your project settings.
⚠ Bitbucket pipelines do not support commit ranges therefore only your latest commit in a pushed group or in a new branch will be scanned.
Configuring a Bitbucket pipeline to use ggshield is as simple as adding a step to your project’s workflow:
Do not forget to add your GitGuardian API Key to the GITGUARDIAN_API_KEY environment variable in your project settings.
Circle CI is supported in ggshield through ggshield-orb.
Do not forget to add your GitGuardian API Key to the GITGUARDIAN_API_KEY environment variable in your project settings.
Do not forget to add your GitGuardian API Key to the GITGUARDIAN_API_KEY environment variable in your project settings.
To add ggshield to your pipelines configure your Jenkinsfile to add a ggshield stage:
Do not forget to add your GitGuardian API Key to the gitguardian-api-key credential in your project settings.
Drone CI integration handles only pull-request or merge-request events, push events are not handled. Do not forget to add your GitGuardian API Key to the GITGUARDIAN_API_KEY environment variable for your Drone CI workers.
⚠ Azure Pipelines does not support commit ranges outside of GitHub Pull Requests, therefore on push events in a regular branch only your latest commit will be scanned. This limitation doesn’t apply to GitHub Pull Requests where all the commits in the pull request will be scanned.
To add ggshield to your pipelines configure your azure-pipelines.yml to add a ggshield scanning job:
Do not forget to add your GitGuardian API Key to the gitguardianApiKey secret variable in your pipeline settings.
If no secrets or policy breaks have been found, the exit code will be 0:
If a secret or other issue is found in your staged code or in your CI, you will have an alert giving you the type of policy break, the filename where the policy break has been found and a patch giving you the position of the policy break in the file:
Related open source projects
GitGuardian shield is MIT licensed.
About
Detect secret in source code, scan your repo for leaks. Find secrets with GitGuardian and prevent leaked credentials. GitGuardian is an automated secrets detection & remediation service.
