Hari Sekhon - DevOps Bash Tools

git.io/bash-tools

550+ DevOps Shell Scripts and Advanced Bash environment.

Fast, Advanced Systems Engineering, Automation, APIs, shorter CLIs, etc.

Heavily used in many GitHub repos, dozens of DockerHub builds (Dockerfiles) and 400+ CI builds.

Summary:

• Scripts for many popular DevOps technologies, see Inventory below for more details
• Advanced configs for common tools like Git, vim, screen, tmux, PostgreSQL psql etc...
• CI configs for most major Continuous Integration products (see CI builds page)
• CI scripts for a drop-in framework of standard checks to run in all CI builds, CI detection, accounting for installation differences across CI environments, root vs user, virtualenvs etc.
• API scripts auto-handling authentication, tokens and other details to quickly query popular APIs with a few keystrokes just supplying the /path/endpoint
• Advanced Bash environment - .bashrc + .bash.d/*.sh - aliases, functions, colouring, dynamic Git & shell behaviour enhancements, automatic pathing for installations and major languages like Python, Perl, Ruby, NodeJS, Golang across Linux distributions and Mac. See .bash.d/README.md
• Installs the best systems packages - AWS CLI, Azure CLI, GCloud SDK, Terraform, Kubernetes kubectl & kustomize, eksctl, jx, jq etc... extensive package lists for servers and desktops for all major Linux distributions package managers and Mac
  • setup/ - contains even more scripts to download and install software, JDBC connectors, Mac OS X settings etc.
• Utility Libraries used by many hundreds of scripts and builds across repos:
  • .bash.d/ - interactive library
  • lib/ - scripting and CI library
• SQL Scripts - 100+ scripts for PostgreSQL, MySQL, AWS Athena + CloudTrail, Google BigQuery
• Templates - templates for common programming languages and build configs
• Kubernetes Configs - Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks

See Also: similar DevOps repos in other languages

Hari Sekhon

Cloud & Big Data Contractor, United Kingdom

(ex-Cloudera, former Hortonworks Consultant)

(you're welcome to connect with me on LinkedIn)

Quick Setup

To bootstrap, install packages and link in to your shell profile to inherit all configs, do:

curl -L https://git.io/bash-bootstrap | sh

• Adds sourcing to .bashrc/.bash_profile to automatically inherit all .bash.d/*.sh environment enhancements for all technologies (see Inventory below)
• Symlinks .* config dotfiles to $HOME for git, vim, top, htop, screen, tmux, editorconfig, Ansible, PostgreSQL .psqlrc etc. (only when they don't already exist so there is no conflict with your own configs)
• Installs OS package dependencies for all scripts (detects the OS and installs the right RPMs, Debs, Apk or Mac HomeBrew packages)
• Installs Python packages including AWS CLI

To only install package dependencies to run scripts, simply cd to the git clone directory and run make:

git clone https://github.com/HariSekhon/DevOps-Bash-tools bash-tools
cd bash-tools
make

make install sets your shell profile to source this repo. See Individual Setup Parts below for more install/uninstall options.

Inventory

• Linux & Mac - curl OAuth / JWT, LDAP, find duplicate files, SSL certificate get/validate, URL encoding/decoding, Vagrant, advanced configurations:
  • .bashrc, .bash.d/*.sh, .gitconfig, .vimrc, .screenrc, .tmux.conf, .toprc, .gitignore...
• AWS - Amazon Web Services - AWS account summary, lots of IAM reports, config hardening, EC2, ECR, EKS, Spot termination, S3 access logging, KMS rotation, SSM, CloudTrail
• GCP - Google Cloud Platform - massive GCP auto-inventory, scripts for GCE, GKE, GCR, Secret Manager, BigQuery, Cloud SQL, Cloud Scheduler, Terraform service account creation
• Kubernetes - massive Kubernetes auto-inventory, cluster management scripts & tricks
• Docker - Docker API, Dockerhub API, Quay.io API scripts
• Databases - fast CLI wrappers, instant Docker sandboxes (PostgreSQL, MySQL, MariaDB, SQLite), SQL scripts, SQL script testers against all versions of a DB, advanced .psqlrc
• Big Data & NoSQL - Kafka, Hadoop, HDFS, Hive, Impala, ZooKeeper, Cloudera Manager API & Cloudera Navigator API scripts
• Git - GitHub, GitLab, Bitbucket, Azure DevOps - scripts for Git local & mirror management, GitHub, GitLab & BitBucket APIs
• CI/CD - Continuous Integration / Continuous Delivery - API scripts & build pipeline configs for just about every major CI system:
  • Jenkins, Concourse, GoCD, TeamCity - one-touch boot & build
  • Azure DevOps Pipelines, GitHub Actions Workflows, GitLab CI, BitBucket Pipelines, AppVeyor, BuildKite, Travis CI, Circle CI, Codefresh, CodeShip, Drone.io, Semaphore CI, Shippable, Wercker ...
• Internet Services - Cloudflare, Pingdom
• Perl - Perl utilities & library management
• Python - Python utilities & library management
• Golang - Golang utilities
• MP3 - MP3 metadata editing, grouping and ordering of albums and audiobooks
• Spotify - 40+ Spotify API scripts for backups, managing playlists, track deduplication, URI conversion, search, add/delete, liked tracks, followed artists, top artists, top tracks etc.
• More Linux & Mac - more systems administration scripts, package installation automation
• Builds, Languages & Linting - programming language, build system & CI linting
• Data Format Conversion & Validation - data validation scripts for CSV, JSON, Avro, Parquet, INI / Properties files (Java), LDAP LDIF, XML, YAML
• Templates - Templates for AWS, GCP, Terraform, Docker, Jenkins, Cloud Build, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml etc.
• Kubernetes Configs - Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks

Linux & Mac:

• .* - dot conf files for lots of common software eg. advanced .vimrc, .gitconfig, massive .gitignore, .editorconfig, .screenrc, .tmux.conf etc.
  • .vimrc - contains many awesome vim tweaks, plus hotkeys for linting lots of different file types in place, including Python, Perl, Bash / Shell, Dockerfiles, JSON, YAML, XML, CSV, INI / Properties files, LDAP LDIF etc without leaving the editor!
  • .screenrc - fancy screen configuration including advanced colour bar, large history, hotkey reloading, auto-blanking etc.
  • .tmux.conf - fancy tmux configuration include advanced colour bar and plugins, settings, hotkey reloading etc.
  • Git:
    • .gitconfig - advanced Git configuration
    • .gitignore - extensive Git ignore of trivial files you shouldn't commit
    • enhanced Git diffs
    • protections against committing AWS secret keys or merge conflict unresolved files
• .bashrc - shell tuning and sourcing of .bash.d/*.sh
• .bash.d/*.sh - thousands of lines of advanced bashrc code, aliases, functions and environment variables for:
  • Linux & Mac
  • SCM - Git, Mercurial, Svn
  • AWS
  • GCP
  • Docker
  • Kubernetes
  • Kafka
  • Vagrant
  • automatic GPG and SSH agent handling for handling encrypted private keys without re-entering passwords, and lazy evaluation to only prompt key load the first time SSH is called
  • and lots more - see .bash.d/README for a more detailed list
  • run make bash to link .bashrc/.bash_profile and the .* dot config files to your $HOME directory to auto-inherit everything
• lib/*.sh - Bash utility libraries full of functions for Docker, environment, CI detection (Travis CI, Jenkins etc), port and HTTP url availability content checks etc. Sourced from all my other GitHub repos to make setting up Dockerized tests easier.
• setup/install_*.sh - various simple to use installation scripts for common technologies like AWS CLI, Azure CLI, GCloud SDK, Terraform, Ansible, MiniKube, MiniShift (Kubernetes / Redhat OpenShift/OKD dev VMs), Maven, Gradle, SBT, EPEL, RPMforge, Homebrew, Travis CI, Circle CI, AppVeyor, BuildKite, Parquet Tools etc.
• clean_caches.sh - cleans out OS package and programming language caches - useful to save space or reduce Docker image size
• curl_auth.sh - shortens curl command by auto-loading your OAuth2 / JWT API token or username & password from environment variables or interactive starred password prompt through a ram file descriptor to avoid placing them on the command line (which would expose your credentials in the process list or OS audit log files). Used by many other adjacent API querying scripts
• ldapsearch.sh - shortens ldapsearch command by inferring switches from environment variables
• ldap_user_recurse.sh / ldap_group_recurse.sh - recurse Active Directory LDAP users upwards to find all parent groups, or groups downwards to find all nested users (useful for debugging LDAP integration and group-based permissions)
• find_duplicate_files*.sh - finds duplicate files by size and/or checksum in given directory trees. Checksums are only done on files that already have matching byte counts for efficiency
• find_broken_links.sh - find broken links with delays to avoid tripping defenses
• jvm_heaps*.sh - show all your Java heap sizes for all running Java processes, and their total MB (for performance tuning and sizing)
• random_select.sh - selects one of given args at random. Useful for sampling, running randomized subsets of large test suites etc.
• split.sh - split large files into N parts (defaults to the number of your CPU cores) to parallelize operations on them
• ssl_get_cert.sh - gets a remote host:port server's SSL cert in a format you can pipe, save and use locally, for example in Java truststores
• ssl_verify_cert.sh - verifies a remote SSL certificate (battle tested more feature-rich version check_ssl_cert.pl exists in the Advanced Nagios Plugins repo)
• urlencode.sh / urldecode.sh - URL encode/decode quickly on the command line, in pipes etc.
• vagrant_hosts.sh - generate /etc/hosts output from a Vagrantfile
• vagrant_total_mb.sh - calculate the RAM committed to VMs in a Vagrantfile

Databases

• sql/ - 100+ SQL scripts for PostgreSQL, MySQL, Google BigQuery and AWS Athena CloudTrail logs integration
• mysql*.sh - MySQL scripts:
  • mysql.sh - shortens mysql command to connect to MySQL by auto-populating switches from both standard environment variables like $MYSQL_TCP_PORT, $DBI_USER, $MYSQL_PWD (see doc) and other common environment variables like $MYSQL_HOST / $HOST, $MYSQL_USER / $USER, $MYSQL_PASSWORD / $PASSWORD, $MYSQL_DATABASE / $DATABASE
  • mysql_foreach_table.sh - executes a SQL query against every table, replacing {db} and {table} in each iteration eg. select count(*) from {table}
  • mysql_*.sh - various scripts using mysql.sh for row counts, iterating each table, or outputting clean lists of databases and tables for quick scripting
  • mysqld.sh - one-touch MySQL, boots docker container + drops in to mysql shell, with /sql scripts mounted in container for easy sourcing eg. source /sql/<name>.sql. Optionally loads sample 'chinook' database
  • see also the SQL Scripts repo for many more straight MySQL SQL scripts
• mariadb.sh - one-touch MariaDB, boots docker container + drops in to mysql shell, with /sql scripts mounted in container for easy sourcing eg. source /sql/<name>.sql. Optionally loads sample 'chinook' database
• sqlite.sh - one-touch SQLite, starts sqlite3 shell with sample 'chinook' database loaded
• postgres*.sh / psql.sh - PostgreSQL scripts:
  • postgres.sh - one-touch PostgreSQL, boots docker container + drops in to psql shell, with /sql scripts mounted in container for easy sourcing eg. \i /sql/<name>.sql. Optionally loads sample 'chinook' database
  • psql.sh - shortens psql command to connect to PostreSQL by auto-populating switches from environment variables, using both standard postgres supported environment variables like $PG* (see doc) as well as other common environment variables like $POSTGRESQL_HOST / $POSTGRES_HOST / $HOST, $POSTGRESQL_USER / $POSTGRES_USER / $USER, $POSTGRESQL_PASSWORD / $POSTGRES_PASSWORD / $PASSWORD, $POSTGRESQL_DATABASE / $POSTGRES_DATABASE / $DATABASE
  • postgres_foreach_table.sh - executes a SQL query against every table, replacing {db}, {schema} and {table} in each iteration eg. select count(*) from {table}
  • postgres_*.sh - various scripts using psql.sh for row counts, iterating each table, or outputting clean lists of databases, schemas and tables for quick scripting

AWS - Amazon Web Services

• aws_*.sh - AWS scripts:
  • aws_account_summary.sh - prints AWS account summary in key = value pairs for easy viewing / grepping of things like AccountMFAEnabled, AccountAccessKeysPresent, useful for checking whether the root account has MFA enabled and no access keys, comparing number of users vs number of MFA devices etc. (see also check_aws_root_account.py in Advanced Nagios Plugins)
  • aws_cloudtrails_cloudwatch.sh - lists Cloud Trails and their last delivery to CloudWatch Logs (should be recent)
  • aws_cloudtrails_event_selectors.sh - lists Cloud Trails and their event selectors to check each one has at least one event selector
  • aws_cloudtrails_s3_accesslogging.sh - lists Cloud Trails buckets and their Access Logging prefix and target bucket. Checks S3 access logging is enabled
  • aws_cloudtrails_s3_kms.sh - lists Cloud Trails and whether their S3 buckets are KMS secured
  • aws_cloudtrails_status.sh - lists Cloud Trails status - if logging, multi-region and log file validation enabled
  • aws_config_all_types.sh - lists AWS Config recorders, checking all resource types are supported (should be true) and includes global resources (should be true)
  • aws_config_recording.sh - lists AWS Config recorders, their recording status (should be true) and their last status (should be success)
  • aws_ecr_tag_image.sh - tags an AWS ECR image with another tag without pulling and pushing it
  • aws_harden_password_policy.sh - strengthens AWS password policy according to CIS Foundations Benchmark recommendations
  • aws_iam_generate_credentials_report_wait.sh - generates an AWS IAM credentials report
  • aws_kms_key_rotation_enabled.sh - lists AWS KMS keys and whether they have key rotation enabled
  • aws_kube_creds.sh - auto-loads all AWS EKS clusters credentials in the current or given --region so your kubectl is ready to rock on AWS
  • aws_meta.sh - AWS EC2 Metadata API query shortcut. See also the official ec2-metadata shell script with more features
  • aws_password_policy.sh - prints AWS password policy in key = value pairs for easy viewing / grepping (used by aws_harden_password_policy.sh before and after to show the differences)
  • aws_policies_attached_to_users.sh - finds AWS IAM policies directly attached to users (anti-best practice) instead of groups
  • aws_policies_granting_full_access.sh - finds AWS IAM policies granting full access (anti-best practice)
  • aws_policies_unattached.sh - lists unattached AWS IAM policies
  • aws_s3_access_logging.sh - lists AWS S3 buckets and their access logging status
  • aws_spot_when_terminated.sh - executes commands when the AWS EC2 instance running this script is notified of Spot Termination, acts as a latch mechanism that can be set any time after boot
  • aws_ssm_put_param.sh - reads a value from a command line argument or non-echo prompt and saves it to AWS Systems Manager Parameter Store. Useful for uploading a password without exposing it on your screen
  • aws_users.sh - list your AWS IAM users
  • aws_users_access_key_age.sh - prints AWS users access key status and age (see also aws_users_access_key_age.py in DevOps Python tools which can filter by age and status)
  • aws_users_access_key_age_report.sh - prints AWS users access key status and age using a bulk credentials report (faster for many users)
  • aws_users_access_key_last_used.sh - prints AWS users access keys last used date
  • aws_users_access_key_last_used_report.sh - same as above using bulk credentials report (faster for many users)
  • aws_users_last_used_report.sh - lists AWS users password/access keys last used dates
  • aws_users_mfa_active_report.sh - lists AWS users password enabled and MFA enabled status
  • aws_users_mfa_serials.sh - lists AWS users MFA serial numbers (differentiates Virtual vs Hardware MFAs)
  • aws_users_pw_last_used.sh - lists AWS users and their password last used date

GCP - Google Cloud Platform

• Google Cloud scripts - gcp_*.sh / gce_*.sh / gke_*.sh / gcr_*.sh / bigquery_*.sh:
  • gcp_terraform_create_credential.sh - creates a service account for Terraform with full permissions, creates and downloads a credential key json and even prints the export GOOGLE_CREDENTIALS command to configure your environment to start using Terraform immediately
  • gcp_info.sh - huge Google Cloud inventory of deployed resources within the current project - Cloud SDK info plus all of the following (detects which services are enabled to query):
    • gcp_info_compute.sh - GCE Virtual Machine instances, App Engine instances, Cloud Functions, GKE clusters, all Kubernetes objects across all GKE clusters (see kubernetes_info.sh below for more details)
    • gcp_info_storage.sh - Cloud SQL info below, plus: Cloud Storage Buckets, Cloud Filestore, Cloud Memorystore Redis, BigTable clusters and instances, Datastore indexes
    • gcp_info_cloud_sql.sh - Cloud SQL instances, whether their backups are enabled, and all databases on each instance
      • gcp_info_cloud_sql_databases.sh - lists databases inside each Cloud SQL instance. Included in gcp_info_cloud_sql.sh
      • gcp_info_cloud_sql_backups.sh - lists backups for each Cloud SQL instance with their dates and status. Not included in gcp_info_cloud_sql.sh for brevity. See also gcp_sql_export.sh further down for more durable backups to GCS
      • gcp_info_cloud_sql_users.sh - lists users for each running Cloud SQL instance. Not included in gcp_info_cloud_sql.sh for brevity but useful to audit users
    • gcp_info_networking.sh - VPC Networks, Addresses, Proxies, Subnets, Routers, Routes, VPN Gateways, VPN Tunnels, Reservations, Firewall rules, Forwarding rules, Cloud DNS managed zones and verified domains
    • gcp_info_bigdata.sh - Dataproc clusters and jobs in all regions, Dataflow jobs in all regions, PubSub messaging topics, Cloud IOT registries in all regions
    • gcp_info_tools.sh - Cloud Source Repositories, Cloud Builds, Container Registry images across all major repos (gcr.io, us.gcr.io, eu.gcr.io, asia.gcr.io), Deployment Manager deployments
    • gcp_info_auth_config.sh - Auth Configurations, Organizations & Current Config
    • gcp_info_projects.sh - Projects names and IDs
    • gcp_info_services.sh - Services & APIs enabled
      • gcp_service_apis.sh - lists all available GCP Services, APIs and their states (enabled/disabled), and provides is_service_enabled() function used throughout the adjacent scripts to avoid errors and only show relevant enabled services
    • gcp_info_accounts_secrets.sh - IAM Service Accounts, Secret Manager secrets
  • gcp_info_all_projects.sh - same as above but for all detected projects
  • gcp_foreach_project.sh - executes a templated command across all GCP projects, switching core/project and replacing {project_id} and {project_name} in each iteration - powerful, use with care! (used by gcp_info_all_projects.sh to call gcp_info.sh)
  • gcp_find_orphaned_disks.sh - lists orphaned disks across all GCP projects (not attached to any compute instance)
  • gcp_secrets_*.sh - Google Secret Manager scripts:
    • gcp_secrets_to_kubernetes.sh - loads GCP secrets to Kubernetes secrets in a 1-to-1 mapping. Can specify a list of secrets or auto-loads all GCP secrets with labels kubernetes-cluster and kubernetes-namespace matching the current kubectl context (kcd to the right namespace first, see .bash.d/kubernetes). See also kubernetes_get_secret_values.sh to debug the actual values that got loaded
    • gcp_secrets_to_kubernetes_multipart.sh - creates a Kubernetes secret from multiple GCP secrets (used to put private.pem and public.pem into the same secret to appear as files on volume mounts for apps in pods to use)
  • gcp_service_account_credential_to_secret.sh - creates GCP service account and exports a credential key to GCP Secret Manager (useful to stage or combine with gcp_secrets_to_kubernetes.sh)
  • gcp_service_accounts_credential_keys.sh - lists all service account credential keys and expiry dates, can grep 9999-12-31T23:59:59Z to find non-expiring keys
  • gcp_service_accounts_credential_keys_age.sh - lists all service account credential keys age in days
  • gcp_service_accounts_credential_keys_expired.sh - lists expired service account credential keys that should be removed and recreated if needed
  • gcr_*.sh - Google Container Registry scripts:
    • gcr_tag_latest.sh - tags a given GCR docker image:tag as latest without pulling or pushing the docker image
    • gcr_tag_datetime.sh - tags a given GCR docker image with its creation date and UTC timestamp (when it was uploaded or created by Google Cloud Build) without pulling or pushing the docker image
    • gcr_newest_image_tags.sh - lists the tags for the given GCR docker image with the newest creation date (can use this to determine which image version to tag as latest)
    • gcr_tag_newest_image_as_latest.sh - finds and tags the newest build of a given GCR docker image as latest without pulling or pushing the docker image
    • gcr_alternate_tags.sh - lists all the tags for a given GCR docker image:tag (use arg <image>:latest to see what version / build hashref / date tag has been tagged as latest)
    • gcr_list_tags.sh - lists all the tags for a given GCR docker image
    • gcr_tags_timestamps.sh - lists all the tags and their timestamps for a given GCR docker image
    • see also cloudbuild.yml in the Templates repo
  • gce_*.sh - Google Compute Engine scripts:
    • gce_meta.sh - simple script to query the GCE metadata API from within Virtual Machines
    • gce_when_preempted.sh - GCE VM preemption latch script - can be executed any time to set one or more commands to execute upon preemption
    • gce_is_preempted.sh - GCE VM return true/false if preempted, callable from other scripts
  • gcp_sql_*.sh - Cloud SQL scripts:
    • gcp_sql_backup.sh - creates Cloud SQL backups
    • gcp_sql_export.sh - creates Cloud SQL exports to GCS
    • gcp_sql_enable_automated_backups.sh - enable automated daily Cloud SQL backups
    • gcp_sql_enable_point_in_time_recovery.sh - enable point-in-time recovery with write-ahead logs
    • gcp_sql_running_primaries.sh - lists primary running Cloud SQL instances
    • gcp_sql_service_accounts.sh - lists Cloud SQL instance service accounts. Useful for copying to IAM to grant permissions (eg. Storage Object Creator for SQL export backups to GCS)
    • gcp_sql_create_readonly_service_account.sh - creates a service account with read-only permissions to Cloud SQL eg. to run export backups to GCS
    • gcp_sql_grant_instances_gcs_object_creator.sh - grants minimal GCS objectCreator permission on a bucket to primary Cloud SQL instances for exports
  • gcp_cloud_schedule_sql_exports.sh - creates Google Cloud Scheduler jobs to trigger a Cloud Function via PubSub to run Cloud SQL exports to GCS for all Cloud SQL instances in the current GCP project
    • the Python GCF function is in the DevOps Python tools repo
  • bigquery_*.sh - BigQuery scripts:
    • bigquery_list_datasets.sh - lists BigQuery datasets in the current GCP project
    • bigquery_list_tables.sh - lists BigQuery tables in a given dataset
    • bigquery_list_tables_all_datasets.sh - lists tables for all datasets in the current GCP project
    • bigquery_foreach_dataset.sh - executes a templated command for each dataset
    • bigquery_foreach_table.sh - executes a templated command for each table in a given dataset
    • bigquery_foreach_table_all_datasets.sh - executes a templated command for each table in each dataset in the current GCP project
    • bigquery_table_row_count.sh - gets the row count for a given table
    • bigquery_tables_row_counts.sh - gets the row counts for all tables in a given dataset
    • bigquery_tables_row_counts_all_datasets.sh - gets the row counts for all tables in all datasets in the current GCP project
    • bigquery_generate_query_biggest_tables_across_datasets_by_row_count.sh - generates a BigQuery SQL query to find the top 10 biggest tables by row count
    • bigquery_generate_query_biggest_tables_across_datasets_by_size.sh - generates a BigQuery SQL query to find the top 10 biggest tables by size
    • see also the SQL Scripts repo for many more straight BigQuery SQL scripts
  • gke_*.sh - Google Kubernetes Engine scripts
    • gke_kube_creds.sh - auto-loads all GKE clusters credentials in the current project so your kubectl is ready to rock on GCP
    • gke_kubectl.sh - runs kubectl commands safely fixed to a given GKE cluster using config isolation to avoid concurrency race conditions
    • gke_persistent_volumes_disk_mappings.sh - lists GKE kubernetes persistent volumes to GCP persistent disk names, along with PVC and namespace, useful when investigating, resizing PVs etc.

Kubernetes

• kubernetes_*.sh - Kubernetes scripts:
  • kubernetes_info.sh - huge Kubernetes inventory listing of deployed resources across all namespaces in the current cluster / kube context:
    • cluster-info
    • master component statuses
    • nodes
    • namespaces
    • deployments, replicasets, replication controllers, statefulsets, daemonsets, horizontal pod autoscalers
    • storage classes, persistent volumes, persistent volume claims
    • service accounts, resource quotas, network policies, pod security policies
    • container images running
    • container images running counts descending
    • pods (might be too much detail if you have high replica counts, so done last, comment if you're sure nobody has deployed pods outside deployments)
  • kubectl.sh - runs kubectl commands safely fixed to a given context using config isolation to avoid concurrency race conditions
  • kubernetes_foreach_context.sh - executes a command across all kubectl contexts, replacing {context} in each iteration (skips lab contexts docker / minikube / minishift to avoid hangs since they're often offline)
  • kubernetes_foreach_namespace.sh - executes a command across all kubernetes namespaces in the current cluster context, replacing {namespace} in each iteration
    • Can be chained with kubernetes_foreach_context.sh and useful when combined with gcp_secrets_to_kubernetes.sh to load all secrets from GCP to Kubernetes for the current cluster, or combined with gke_kube_creds.sh and kubernetes_foreach_context.sh for all clusters! Powerful stuff, use with care.
  • kubeadm_join_cmd.sh - outputs kubeadm join command (generates new token) to join an existing Kubernetes cluster (used in vagrant kubernetes provisioning scripts)
  • kubeadm_join_cmd2.sh - outputs kubeadm join command manually (calculates cert hash + generates new token) to join an existing Kubernetes cluster
  • kubectl_exec.sh - finds and execs to the first Kubernetes pod matching given pod filters, optionally specifying the container to exec to, and shows the full generated kubectl exec command line for clarity
  • kubectl_exec_grep.sh - finds and execs to the first Kubernetes pod matching the given name regex, optionally specifying the container name regex to exec to, and shows the full generated kubectl exec command line for clarity
  • kubernetes_api.sh - finds Kubernetes API and runs your curl arguments against it, auto-getting authorization token and auto-populating OAuth authentication header
  • kubernetes_etcd_backup.sh - creates a timestamped backup of the Kubernetes Etcd database for a kubeadm cluster
  • kubernetes_pods_per_nodes.sh - lists number of pods per node sorted descending
  • kubernetes_find_stuck_jobs.sh - finds Kubernetes jobs stuck for hours or days with no completions
  • kubernetes_delete_stuck_jobs.sh - prompts for confirmation to delete stuck Kubernetes jobs found by script above
  • kubernetes_running_images.sh - lists Kubernetes container images running on the current cluster
  • kubernetes_running_image_counts.sh - lists Kubernetes container images running counts sorted descending
  • kubernetes_get_secret_values.sh - prints the keys and base64 decoded values within a given Kubernetes secret for quick debugging of Kubernetes secrets. See also: gcp_secrets_to_kubernetes.sh
  • see also the Kubernetes templates repo

Docker

• docker_*.sh / dockerhub_*.sh - Docker / DockerHub API scripts:
  • dockerhub_api.sh - queries DockerHub API v2 with or without authentication ($DOCKERHUB_USER & $DOCKERHUB_PASSWORD / $DOCKERHUB_TOKEN)
  • docker_api.sh - queries a Docker Registry with optional basic authentication if $DOCKER_USER & $DOCKER_PASSWORD are set
  • docker_registry_list_images.sh - lists images in a given private Docker Registry
  • docker_registry_list_tags.sh - lists tags for a given image in a private Docker Registry
  • docker_registry_get_image_manifest.sh - gets a given image:tag manifest from a private Docker Registry
  • docker_registry_tag_image.sh - tags a given image with a new tag in a private Docker Registry via the API without pulling and pushing the image data (must faster and more efficient)
  • dockerhub_list_tags.sh - lists tags for a given DockerHub repo. See also dockerhub_show_tags.py in the DevOps Python tools repo.
  • dockerhub_list_tags_by_last_updated.sh - lists tags for a given DockerHub repo sorted by last updated timestamp descending
  • dockerhub_search.sh - searches with a configurable number of returned items (older docker cli was limited to 25 results)
  • clean_caches.sh - cleans out OS package and programming language caches, call near end of Dockerfile to reduce Docker image size
  • see also the Dockerfiles repo
• quay.io_api.sh - queries the Quay.io API with OAuth2 authentication token $QUAY_TOKEN

Big Data & NoSQL

• kafka_*.sh - scripts to make Kafka CLI usage easier including auto-setting Kerberos to source TGT from environment and auto-populating broker and zookeeper addresses. These are auto-added to the $PATH when .bashrc is sourced. For something similar for Solr, see solr_cli.pl in the DevOps Perl Tools repo.
• zookeeper*.sh - Apache ZooKeeper scripts:
  • zookeeper_client.sh - shortens zookeeper-client command by auto-populating the zookeeper quorum from the environment variable $ZOOKEEPERS or else parsing the zookeeper quorum from /etc/**/*-site.xml to make it faster and easier to connect
  • zookeeper_shell.sh - shortens Kafka's zookeeper-shell command by auto-populating the zookeeper quorum from the environment variable $KAFKA_ZOOKEEPERS and optionally $KAFKA_ZOOKEEPER_ROOT to make it faster and easier to connect
• hive_*.sh / beeline*.sh - Apache Hive scripts:
  • beeline.sh - shortens beeline command to connect to HiveServer2 by auto-populating Kerberos and SSL settings, zookeepers for HiveServer2 HA discovery if the environment variable $HIVE_HA is set or using the $HIVESERVER_HOST environment variable so you can connect with no arguments (prompts for HiveServer2 address if you haven't set $HIVESERVER_HOST or $HIVE_HA)
    • beeline_zk.sh - same as above for HiveServer2 HA by auto-populating SSL and ZooKeeper service discovery settings (specify $HIVE_ZOOKEEPERS environment variable to override). Automatically called by beeline.sh if either $HIVE_ZOOKEEPERS or $HIVE_HA is set (the latter parses hive-site.xml for the ZooKeeper addresses)
  • hive_foreach_table.sh - executes a SQL query against every table, replacing {db} and {table} in each iteration eg. select count(*) from {table}
  • hive_list_databases.sh - list Hive databases, one per line, suitable for scripting pipelines
  • hive_list_tables.sh - list Hive tables, one per line, suitable for scripting pipelines
  • hive_tables_metadata.sh - lists a given DDL metadata field for each Hive table (to compare tables)
  • hive_tables_location.sh - lists the data location per Hive table (eg. compare external table locations)
  • hive_tables_row_counts.sh - lists the row count per Hive table
  • hive_tables_column_counts.sh - lists the column count per Hive table
• impala*.sh - Apache Impala scripts:
  • impala_shell.sh - shortens impala-shell command to connect to Impala by parsing the Hadoop topology map and selecting a random datanode to connect to its Impalad, acting as a cheap CLI load balancer. For a real load balancer see HAProxy config for Impala (and many other Big Data & NoSQL technologies). Optional environment variables $IMPALA_HOST (eg. point to an explicit node or an HAProxy load balancer) and IMPALA_SSL=1 (or use regular impala-shell --ssl argument pass through)
  • impala_foreach_table.sh - executes a SQL query against every table, replacing {db} and {table} in each iteration eg. select count(*) from {table}
  • impala_list_databases.sh - list Impala databases, one per line, suitable for scripting pipelines
  • impala_list_tables.sh - list Impala tables, one per line, suitable for scripting pipelines
  • impala_tables_metadata.sh - lists a given DDL metadata field for each Impala table (to compare tables)
  • impala_tables_location.sh - lists the data location per Impala table (eg. compare external table locations)
  • impala_tables_row_counts.sh - lists the row count per Impala table
  • impala_tables_column_counts.sh - lists the column count per Impala table
• hdfs_*.sh - Hadoop HDFS scripts:
  • hdfs_checksum*.sh - walks an HDFS directory tree and outputs HDFS native checksums (faster) or portable externally comparable CRC32, in serial or in parallel to save time
  • hdfs_find_replication_factor_1.sh / hdfs_set_replication_factor_3.sh - finds HDFS files with replication factor 1 / sets HDFS files with replication factor <=2 to replication factor 3 to repair replication safety and avoid no replica alarms during maintenance operations (see also Python API version in the DevOps Python Tools repo)
  • hdfs_file_size.sh / hdfs_file_size_including_replicas.sh - quickly differentiate HDFS files raw size vs total replicated size
  • hadoop_random_node.sh - picks a random Hadoop cluster worker node, like a cheap CLI load balancer, useful in scripts when you want to connect to any worker etc. See also the read HAProxy Load Balancer configurations which focuses on master nodes
• cloudera_*.sh - Cloudera scripts:
  • cloudera_manager_api.sh - script to simplify querying Cloudera Manager API using environment variables, prompts, authentication and sensible defaults. Built on top of curl_auth.sh
  • cloudera_manager_impala_queries*.sh - queries Cloudera Manager for recent Impala queries, failed queries, exceptions, DDL statements, metadata stale errors, metadata refresh calls etc. Built on top of cloudera_manager_api.sh
  • cloudera_manager_yarn_apps.sh - queries Cloudera Manager for recent Yarn apps. Built on top of cloudera_manager_api.sh
  • cloudera_navigator_api.sh - script to simplify querying Cloudera Navigator API using environment variables, prompts, authentication and sensible defaults. Built on top of curl_auth.sh
  • cloudera_navigator_audit_logs.sh - fetches Cloudera Navigator audit logs for given service eg. hive/impala/hdfs via the API, simplifying date handling, authentication and common settings. Built on top of cloudera_navigator_api.sh
  • cloudera_navigator_audit_logs_download.sh - downloads Cloudera Navigator audit logs for each service by year. Skips existing logs, deletes partially downloaded logs on failure, generally retry safe (while true, Control-C, not kill -9 obviously). Built on top of cloudera_navigator_audit_logs.sh

Git - GitHub, GitLab, Bitbucket, Azure DevOps

• git*.sh - Git scripts:
  • git_foreach_branch.sh - executes a command on all branches (useful in heavily version branched repos like in my Dockerfiles repo)
  • git_foreach_repo.sh - executes a command against all adjacent repos from a given repolist (used heavily by many adjacent scripts)
  • git_foreach_modified.sh - executes a command against each file with git modified status
  • git_merge_all.sh / git_merge_master.sh / git_merge_master_pull.sh - merges updates from master branch to all other branches to avoid drift on longer lived feature branches / version branches (eg. Dockerfiles repo)
  • git_remotes_add_public_repos.sh - auto-creates remotes for the 4 major public repositories (GitHub/GitLab/Bitbucket/Azure DevOps), useful for git pull -all to fetch and merge updates from all providers in one command
  • git_remotes_set_multi_origin.sh - sets up multi-remote origin for unified push to automatically keep the 4 major public repositories in sync (especially useful for Bitbucket and Azure DevOps which don't have GitLab's auto-mirroring from GitHub feature)
  • git_remotes_set_ssh_to_https.sh - converts local repo's remote URLs from ssh to https (to get through corporate firewalls), auto-loads http auth tokens if found in environment variables
  • git_remotes_set_https_to_ssh.sh - converts local repo's remote URLs from https to ssh (more convenient with SSH keys instead of http auth tokens)
  • git_repos_pull.sh - pull multiple repos based on a source file mapping list - useful for easily sync'ing lots of Git repos among computers
  • git_repos_update.sh - same as above but also runs the make update build to install the latest dependencies, leverages the above script
  • git_log_empty_commits.sh - find empty commits in git history (eg. if a git filter-branch was run but --prune-empty was forgotten, leaking metadata like subjects containing file names or other sensitive info)
  • git_filter_branch_fix_author.sh - rewrites Git history to replace author/committer name & email references (useful to replace default account commits). Powerful, read --help and man git-filter-branch carefully. Should only be used by Git Experts.
  • git_submodules_update_repos.sh - updates submodules (pulls and commits latest upstream github repo submodules) - used to cascade submodule updates throughout all my repos
• github_*.sh - GitHub API scripts:
  • github_api.sh - queryies the GitHub API. Can infer GitHub user, repo and authentication token from local checkout or environment ($GITHUB_USER, $GITHUB_TOKEN)
  • github_foreach_repo.sh - executes a templated command for each non-fork GitHub repo, replacing the {user} and {repo} in each iteration
  • github_actions_runner.sh - downloads, configures and runs a local GitHub Actions Runner
  • github_runners.sh - lists GitHub Actions runners
  • github_workflows.sh - lists GitHub Actions workflows for a given repo (or auto-infers local repository)
  • github_workflow_runs.sh - lists GitHub Actions workflow runs for a given workflow id or name
  • github_workflows_status.sh - lists all GitHub Actions workflows and their statuses for a given repo
  • github_get_user_ssh_public_keys.sh - fetches a given GitHub user's public SSH keys via the API for piping to ~/.ssh/authorized_keys or adjacent tools
  • github_get_ssh_public_keys.sh - fetches the currently authenticated GitHub user's public SSH keys via the API, similar to above but authenticated to get identifying key comments
  • github_add_ssh_public_keys.sh - uploads SSH keys from local files or standard input to the currently authenticated GitHub account. Specify pubkey files (default: ~/.ssh/id_rsa.pub) or read from standard input for piping from adjacent tools
  • github_delete_ssh_public_keys.sh - deletes given SSH keys from the currently authenticated GitHub account by key id or title regex match
  • github_generate_status_page.sh - generates a STATUS.md page by merging all the README.md headers for all of a user's non-forked GitHub repos or a given list of any repos etc.
  • github_sync_repo_descriptions.sh - syncs GitHub repo descriptions to GitLab & BitBucket repos
  • github_repos_sync_status.sh - determines whether each GitHub repo's mirrors on GitLab / BitBucket are up to date with the latest commits, by querying all 3 APIs and comparing master branch hashrefs
• gitlab_*.sh - GitLab API scripts:
  • gitlab_api.sh - queries the GitLab API. Can infer GitLab user, repo and authentication token from local checkout or environment ($GITLAB_USER, $GITLAB_TOKEN)
  • gitlab_foreach_repo.sh - executes a templated command for each GitLab project/repo, replacing the {user} and {project} in each iteration
  • gitlab_project_mirrors.sh - lists each GitLab repo and whether it is a mirror or not
  • gitlab_pull_mirror.sh - trigger a GitLab pull mirroring for a given project's repo, or auto-infers project name from the local git repo
  • gitlab_set_project_description.sh - sets the description for one or more projects using the GitLab API
  • gitlab_get_user_ssh_public_keys.sh - fetches a given GitLab user's public SSH keys via the API, with identifying comments, for piping to ~/.ssh/authorized_keys or adjacent tools
  • gitlab_get_ssh_public_keys.sh - fetches the currently authenticated GitLab user's public SSH keys via the API
  • gitlab_add_ssh_public_keys.sh - uploads SSH keys from local files or standard input to the currently authenticated GitLab account. Specify pubkey files (default: ~/.ssh/id_rsa.pub) or read from standard input for piping from adjacent tools
  • gitlab_delete_ssh_public_keys.sh - deletes given SSH keys from the currently authenticated GitLab account by key id or title regex match
  • gitlab_validate_ci_yaml.sh - validates a .gitlab-ci.yml file via the GitLab API
• bitbucket_*.sh - BitBucket API scripts:
  • bitbucket_api.sh - queries the BitBucket API. Can infer BitBucket user, repo and authentication token from local checkout or environment ($BITBUCKET_USER, $BITBUCKET_TOKEN)
  • bitbucket_foreach_repo.sh - executes a templated command for each BitBucket repo, replacing the {user} and {repo} in each iteration
  • bitbucket_set_project_description.sh - sets the description for one or more repos using the BitBucket API
  • bitbucket_get_ssh_public_keys.sh - fetches the currently authenticated BitBucket user's public SSH keys via the API for piping to ~/.ssh/authorized_keys or adjacent tools
  • bitbucket_add_ssh_public_keys.sh - uploads SSH keys from local files or standard input to the currently authenticated BitBucket account. Specify pubkey files (default: ~/.ssh/id_rsa.pub) or read from standard input for piping from adjacent tools

CI/CD - Continuous Integration / Continuous Deployment

• azure_devops_api.sh - queries Azure DevOps's API with authentication
• jenkins_*.sh - Jenkins CI scripts:
  • jenkins_cli.sh - shortens jenkins-cli.jar command by auto-inferring basic configuations, auto-downloading the CLI if absent, inferrings a bunch of Jenkins related variables like $JENKINS_URL and authentication from $JENKINS_USER/$JENKINS_PASSWORD, or finds admin password from inside local docker container. Used heavily by jenkins.sh one-shot setup
  • jenkins_password.sh - gets Jenkins admin password from local docker container. Used by jenkins_cli.sh
  • jenkins.sh - one-touch Jenkins CI, launches in docker, installs plugins, validates Jenkinsfile, configures jobs from $PWD/setup/jenkins-job.xml and sets Pipeline to git remote origin's Jenkinsfile, triggers build, tails results in terminal. Call from any repo top level directory with a Jenkinsfile pipeline and setup/jenkins-job.xml (all mine have it)
• concourse.sh - one-touch Concourse CI, launches in docker, configures pipeline from $PWD/.concourse.yml, triggers build, tails results in terminal, prints recent build statuses at end. Call from any repo top level directory with a .concourse.yml config (all mine have it), mimicking structure of fully managed CI systems
  • fly.sh - shortens fly command to not have to specify target all the time
• gocd.sh - one-touch GoCD CI, launches in docker, (re)creates config repo ($PWD/setup/gocd_config_repo.json) from which to source pipeline(s) (.gocd.yml), detects and enables agent(s) to start building. Call from any repo top level directory with a .gocd.yml config (all mine have it), mimicking structure of fully managed CI systems
• gocd_api.sh - queries the GoCD API
• teamcity_*.sh - TeamCity CI API scripts:
  • teamcity.sh - boots TeamCity CI cluster in docker, just click proceed and accept the EULA and it does the rest, it even creates an admin user and an API token for you
  • See Also: TeamCity CI config repo
  • teamcity_api.sh - queries TeamCity's API, auto-handling authentication and other quirks of the API
  • teamcity_create_project.sh - creates a TeamCity project using the API
  • teamcity_create_github_oauth_connection.sh - creates a TeamCity GitHub OAuth VCS connection in the Root project, useful for bootstrapping projects from VCS configs
  • teamcity_create_vcs_root.sh - creates a TeamCity VCS root from a save configuration (XML or JSON), as downloaded by teamcity_vcs_roots_download.sh
  • teamcity_agents.sh - lists TeamCity agents, their connected state, authorized state, whether enabled and up to date
  • teamcity_builds.sh - lists the last 100 TeamCity builds along with the their state (eg. finished) and status (eg. SUCCESS/FAILURE)
  • teamcity_buildtypes.sh - lists TeamCity buildTypes (pipelines) along with the their project and IDs
  • teamcity_buildtype_create.sh - creates a TeamCity buildType from a local JSON configuration (see teamcity_buildtypes_download.sh)
  • teamcity_buildtype_set_description_from_github.sh - sync's a TeamCity buildType's description from its Github repo description
  • teamcity_buildtypes_set_description_from_github.sh - sync's all TeamCity buildType descriptions from their GitHub repos where available
  • teamcity_export.sh - downloads TeamCity configs to local JSON files in per-project directories mimicking native TeamCity directory structure and file naming
  • teamcity_export_project_config.sh - downloads TeamCity project config to local JSON files
  • teamcity_export_buildtypes.sh - downloads TeamCity buildType config to local JSON files
  • teamcity_export_vcs_roots.sh - downloads TeamCity VCS root config to local JSON files
  • teamcity_projects.sh - lists TeamCity project IDs and Names
  • teamcity_project_set_versioned_settings.sh - configures a project to track all changes to a VCS (eg. GitHub)
  • teamcity_project_vcs_versioning.sh - quickly toggle VCS versioning on/off for a given TeamCity project (useful for testing without auto-committing)
  • teamcity_vcs_roots.sh - lists TeamCity VCS root IDs and Names
• travis_*.sh - Travis CI API scripts (one of my all-time favourite CI systems):
  • travis_api.sh - queries the Travis CI API with authentication using $TRAVIS_TOKEN
  • travis_repos.sh - lists Travis CI repos
  • travis_foreach_repo.sh - executes a templated command against all Travis CI repos
  • travis_repo_build.sh - triggers a build for the given repo
  • travis_repo_caches.sh - lists caches for a given repo
  • travis_repo_crons.sh - lists crons for a given repo
  • travis_repo_env_vars.sh - lists environment variables for a given repo
  • travis_repo_settings.sh - lists settings for a given repo
  • travis_repo_create_cron.sh - creates a cron for a given repo and branch
  • travis_repo_delete_crons.sh - deletes all crons for a given repo
  • travis_repo_delete_caches.sh - deletes all caches for a given repo (sometimes clears build problems)
  • travis_delete_cron.sh - deletes a Travis CI cron by ID
  • travis_repos_settings.sh - lists settings for all repos
  • travis_repos_caches.sh - lists caches for all repos
  • travis_repos_crons.sh - lists crons for all repos
  • travis_repos_create_cron.sh - creates a cron for all repos
  • travis_repos_delete_crons.sh - deletes all crons for all repos
  • travis_repos_delete_caches.sh - deletes all caches for all repos
  • travis_lint.sh - lints a given .travis.yml using the API
• buildkite_*.sh - BuildKite API scripts:
  • buildkite_pipelines.sh - list buildkite pipelines for your $BUILDKITE_ORGANIZATION / $BUILDKITE_USER
  • buildkite_foreach_pipeline.sh - executes a templated command for each Buildkite pipeline, replacing the {user} and {pipeline} in each iteration
  • buildkite_agent.sh - runs a buildkite agent locally on Linux or Mac, or in Docker with choice of Linux distros
  • buildkite_agents.sh - lists the Buildkite agents connected along with their hostname, IP, started dated and agent details
  • buildkite_pipelines.sh - lists Buildkite pipelines
  • buildkite_create_pipeline.sh - create a Buildkite pipeline from a JSON configuration (like from buildkite_get_pipeline.sh or buildkite_save_pipelines.sh)
  • buildkite_get_pipeline.sh - gets details for a specific Buildkite pipeline in JSON format
  • buildkite_cancel_scheduled_builds.sh - cancels BuildKite scheduled builds (to clear a backlog due to offline agents and just focus on new builds)
  • buildkite_rebuild_cancelled_builds.sh - triggers rebuilds of any cancelled pipelines
  • buildkite_rebuild_failed_builds.sh - triggers rebuilds of any failed pipelines (useful if you killed an agent and want to re-run them)
  • buildkite_recreate_pipeline.sh - recreates a pipeline to wipe out all stats (see url and badge caveats in --help)
  • buildkite_running_builds.sh - lists running builds and the agent they're running on
  • buildkite_save_pipelines.sh - saves all BuildKite pipelines in your $BUILDKITE_ORGANIZATION to local JSON files in $PWD/.buildkite-pipelines/
  • buildkite_trigger.sh - triggers BuildKite build job for a given pipeline
  • buildkite_trigger_all.sh - same as above but for all pipelines
• appveyor_api.sh - queries AppVeyor's API with authentication
• codeship_api.sh - queries CodeShip's API with authentication
• drone_api.sh - queries Drone.io's API with authentication
• shippable_api.sh - queries Shippable's API with authentication
• wercker_app_api.sh - queries Wercker's Applications API with authentication

Internet Services

• atlassian_cidr_ranges.sh - lists Atlassian's IPv4 and/or IPv6 cidr ranges via its API
• cloudflare_*.sh - Cloudflare API queries and reports:
  • cloudflare_api.sh - queries the Cloudflare API, handling authentication from $CLOUDFLARE_TOKEN
  • cloudflare_cidr_ranges.sh - lists Cloudflare's IPv4 and/or IPv6 cidr ranges via its API
  • cloudflare_custom_certificates.sh - lists any custom SSL certificates in a given Cloudflare zone along with their status and expiry date
  • cloudflare_dns_records.sh - lists any Cloudflare DNS records for a zone, including the type and ttl
  • cloudflare_dns_records_all_zones.sh - same as above but for all zones
  • cloudflare_dnssec.sh - lists the Cloudflare DNSSec status for all zones
  • cloudflare_foreach_account.sh - executes a templated command for each Cloudflare account, replacing the {account_id} and {account_name} in each iteration (useful for chaining with cloudflare_api.sh)
  • cloudflare_foreach_zone.sh - executes a templated command for each Cloudflare zone, replacing the {zone_id} and {zone_name} in each iteration (useful for chaining with cloudflare_api.sh, used by adjacent cloudflare_*_all_zones.sh scripts)
  • cloudflare_ssl_verified.sh - gets the Cloudflare zone SSL verification status for a given zone
  • cloudflare_ssl_verified_all_zones.sh - same as above for all zones
• pingdom_*.sh - Pingdom API queries and reports for status, latency, average response times, latency averages by hour, SMS credits, outages periods and durations over the last year etc.
  • pingdom_api.sh - Solarwinds Pingdom API query script
  • pingdom_foreach_check.sh - executes a templated command against each Pingdom check, replacing the {check_id} and {check_name} in each iteration
  • pingdom_checks.sh - show all Pingdom checks, status and latencies
  • pingdom_checks_outages.sh / pingdom_checks_outages.sh - show one or all Pingdom checks outage histories for the last year
  • pingdom_checks_average_response_times.sh - shows the average response times for all Pingdom checks for the last week
  • pingdom_check_latency_by_hour.sh / pingdom_checks_latency_by_hour.sh - shows the average latency for one or all Pingdom checks broken down by hour of the day, over the last week
  • pingdom_sms_credits.sh - gets the remaining number of Pingdom SMS credits

Perl

• perl_cpanm_install.sh - bulk installs CPAN modules from mix of arguments / file lists / stdin, accounting for User vs System installs, root vs user sudo, Perlbrew / Google Cloud Shell environments, Mac vs Linux library paths, ignore failure option, auto finds and reads build failure log for quicker debugging showing root cause error in CI builds logs etc
• perl_cpanm_install_if_absent.sh - installs CPAN modules not already in Perl libary path (OS or CPAN installed) for faster installations only where OS packages are already providing some of the modules, reducing time and failure rates in CI builds
• perlpath.sh - prints all Perl libary search paths, one per line
• perl_find_library_path.sh - finds directory where a CPAN module is installed - without args finds the Perl library base
• perl_find_library_executable.sh - finds directory where a CPAN module's CLI program is installed (system vs user, useful when it gets installed to a place that isn't in your $PATH, where which won't help)
• perl_find_unused_cpan_modules.sh - finds CPAN modules that aren't used by any programs in the current directory tree
• perl_find_duplicate_cpan_requirements.sh - finds duplicate CPAN modules listed for install more than once under the directory tree (useful for deduping module installs in a project and across submodules)
• perl_generate_fatpacks.sh - creates Fatpacks - self-contained Perl programs with all CPAN modules built-in

Python

• python_compile.sh - byte-compiles Python scripts and libraries into .pyo optimized files
• python_pip_install.sh - bulk installs PyPI modules from mix of arguments / file lists / stdin, accounting for User vs System installs, root vs user sudo, VirtualEnvs / Anaconda / GitHub Workflows/ Google Cloud Shell, Mac vs Linux library paths, and ignore failure option
• python_pip_install_if_absent.sh - installs PyPI modules not already in Python libary path (OS or pip installed) for faster installations only where OS packages are already providing some of the modules, reducing time and failure rates in CI builds
• python_pip_reinstall_all_modules.sh - reinstalls all PyPI modules which can fix some issues
• pythonpath.sh - prints all Python libary search paths, one per line
• python_find_library_path.sh - finds directory where a PyPI module is installed - without args finds the Python library base
• python_find_library_executable.sh - finds directory where a PyPI module's CLI program is installed (system vs user, useful when it gets installed to a place that isn't in your $PATH, where which won't help)
• python_find_unused_pip_modules.sh - finds PyPI modules that aren't used by any programs in the current directory tree
• python_find_duplicate_pip_requirements.sh - finds duplicate PyPI modules listed for install under the directory tree (useful for deduping module installs in a project and across submodules)
• python_module_to_import_name.sh - converts PyPI module names to Python import names, used by python_find_unused_pip_modules.sh
• python_pyinstaller.sh - creates PyInstaller self-contained Python programs with Python interpreter and all PyPI modules included
• python_pypi_versions.sh - prints all available versions of a given PyPi module using the API

Golang

• golang_get_install.sh - bulk installs Golang modules from mix of arguments / file lists / stdin
• golang_get_install_if_absent.sh - same as above but only if the package binary isn't already available in $PATH
• golang_rm_binaries.sh - deletes binaries of the same name adjacent to .go files. Doesn't delete you bin/ etc as these are often real deployed applications rather than development binaries

MP3

• mp3_set_artist.sh / mp3_set_album.sh - sets the artist / album tag for all mp3 files under given directories. Useful for grouping artists/albums and audiobook author/books (eg. for correct importing into Mac's Books.app)
• mp3_set_track_name.sh - sets the track name metadata for mp3 files under given directories to follow their filenames. Useful for correctly displaying audiobook progress / chapters etc.
• mp3_set_track_order.sh - sets the track order metadata for mp3 files under given directories to follow the lexical file naming order. Useful for correctly ordering album songs and audiobook chapters (eg. for Mac's Books.app). Especially useful for enforcing global ordering on multi-CD audiobooks after grouping into a single audiobook using mp3_set_album.sh (otherwise default track numbers in each CD interleave in Mac's Books.app)

Spotify

40+ Spotify API scripts (used extensively to manage my Spotify-Playlists repo):

• spotify_playlists*.sh - list playlists in either <id> <name> or JSON format
• spotify_playlist_tracks*.sh - gets playlist contents as track URIs / Artists - Track / CSV format - useful for backups or exports between music systems
• spotify_backup.sh - backup all Spotify playlists as well as the ordered list of playlists
• spotify_backup_playlist*.sh - backup Spotify playlists to local files in both human readable Artist - Track format and Spotify URI format for easy restores or adding to new playlists
• spotify_search*.sh - search Spotify's library for tracks / albums / artists getting results in human readable format, JSON, or URI formats for easy loading to Spotify playlists
• spotify_release_year.sh - searches for a given track or album and finds the original release year
• spotify_uri_to_name.sh - convert Spotify track / album / artist URIs to human readable Artist - Track / CSV format. Takes Spotify URIs, URL links or just IDs. Reads URIs from files or standard input
• spotify_create_playlist.sh - creates a Spotify playlist, either public or private
• spotify_rename_playlist.sh - renames a Spotify playlist
• spotify_set_playlists_public.sh / spotify_set_playlists_private.sh - sets one or more given Spotify playlists to public / private
• spotify_add_to_playlist.sh - adds tracks to a given playlist. Takes a playlist name or ID and Spotify URIs in any form from files or standard input. Can be combined with many other tools listed here which output Spotify URIs, or appended from other playlists. Can also be used to restore a spotify playlist from backups
• spotify_delete_from_playlist.sh - deletes tracks from a given playlist. Takes a playlist name or ID and Spotify URIs in any form from files or standard input, optionally prefixed with a track position to remove only specific occurrences (useful for removing duplicates from playlists)
• spotify_delete_from_playlist_if_in_other_playlists.sh - deletes tracks from a given playlist if their URIs are found in the subsequently given playlists
• spotify_duplicate_uri_in_playlist.sh - finds duplicate Spotify URIs in a given playlist (these are guaranteed exact duplicate matches), returns all but the first occurrence and optionally their track positions (zero-indexed to align with the Spotify API for easy chaining with other tools)
• spotify_duplicate_tracks_in_playlist.sh - finds duplicate Spotify tracks in a given playlist (these are idential Artist - Track name matches, which may be from different albums / singles)
• spotify_delete_duplicates_in_playlist.sh - deletes duplicate Spotify URI tracks (identical) in a given playlist using spotify_duplicate_uri_in_playlist.sh and spotify_delete_from_playlist.sh
• spotify_delete_duplicate_tracks_in_playlist.sh - deletes duplicate Spotify tracks (name matched) in a given playlist using spotify_duplicate_tracks_in_playlist.sh and spotify_delete_from_playlist.sh
• spotify_delete_any_duplicates_in_playlist.sh - calls both of the above scripts to first get rid of duplicate URIs and then remove any other duplicates by track name matches
• spotify_playlist_tracks_uri_in_year.sh - finds track URIs in a playlist where their original release date is in a given year or decade (by regex match)
• spotify_playlist_uri_offset.sh - finds the offset of a given track URI in a given playlist, useful to find positions to resume processing a large playlist
• spotify_top_artists*.sh - lists your top artists in URI or human readable format
• spotify_top_tracks*.sh - lists top tracks in URI or human readable format
• spotify_liked_tracks*.sh - lists your Liked Songs in URI or human readable formats
• spotify_liked_artists*.sh - list artists from Liked Songs in URI or human readable formats
• spotify_artists_followed*.sh - lists all followed artists in URI or human readable formats
• spotify_follow_artists.sh - follows artists for the given URIs from files or standard input
• spotify_follow_liked_artists.sh - follows artists with N or more tracks in your Liked Songs
• spotify_set_tracks_uri_to_liked.sh - sets a list of spotify track URIs to 'Liked' so they appear in the Liked Songs playlist. Useful for marking all the tracks in your best playlists as favourite tracks, or for porting historical Starred tracks to the newer Liked Songs
• spotify_foreach_playlist.sh - executes a templated command against all playlists, replacing {playlist} and {playlist_id} in each iteration
• spotify_playlist_name_to_id.sh / spotify_playlist_id_to_name.sh - convert playlist names <=> IDs
• spotify_api_token.sh - gets a Spotify authentication token using either Client Credentials or Authorization Code authentication flows, the latter being able to read/modify private user data, automatically used by spotify_api.sh
• spotify_api.sh - query any Spotify API endpoint with authentication, used by adjacent spotify scripts

More Linux & Mac:

• Linux / Mac systems administration scripts:
  • installation scripts for various OS packages (RPM, Deb, Apk) for various Linux distros (Redhat RHEL / CentOS / Fedora, Debian / Ubuntu, Alpine)
  • install if absent scripts for Python, Perl, Ruby, NodeJS and Golang packages - good for minimizing the number of source code installs by first running the OS install scripts and then only building modules which aren't already detected as installed (provided by system packages), speeding up builds and reducing the likelihood of compile failures
  • install scripts for Jython and build tools like Gradle and SBT for when Linux distros don't provide packaged versions or where the packaged versions are too old
  • OS / Distro Package Management:
    • install_packages.sh - installs package lists from arguments, files or stdin on major linux distros and Mac, detecting the package manager and invoking the right install commands, with sudo if not root. Works on RHEL / CentOS / Fedora, Debian / Ubuntu, Alpine, and Mac Homebrew. Leverages and supports all features of the distro / OS specific install scripts listed below
    • install_packages_if_absent.sh - installs package lists if they're not already installed, saving time and minimizing install logs / CI logs, same support list as above
    • Redhat RHEL / CentOS:
      • yum_install_packages.sh / yum_remove_packages.sh - installs RPM lists from arguments, files or stdin. Handles Yum + Dnf behavioural differences, calls sudo if not root, auto-attempts variations of python/python2/python3 package names. Avoids yum slowness by checking if rpm is installed before attempting to install it, accepts NO_FAIL=1 env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across RHEL/CentOS/Fedora versions)
      • yum_install_packages_if_absent.sh - installs RPMs only if not already installed and not a metapackage provided by other packages (eg. vim metapackage provided by vim-enhanced), saving time and minimizing install logs / CI logs, plus all the features of yum_install_packages.sh above
      • rpms_filter_installed.sh / rpms_filter_not_installed.sh - pipe filter packages that are / are not installed for easy script piping
    • Debian / Ubuntu:
      • apt_install_packages.sh / apt_remove_packages.sh - installs Deb package lists from arguments, files or stdin. Auto calls sudo if not root, accepts NO_FAIL=1 env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across Debian/Ubuntu distros/versions)
      • apt_install_packages_if_absent.sh - installs Deb packages only if not already installed, saving time and minimizing install logs / CI logs, plus all the features of apt_install_packages.sh above
      • apt_wait.sh - blocking wait on concurrent apt locks to avoid failures and continue when available, mimicking yum's waiting behaviour rather than error'ing out
      • debs_filter_installed.sh / debs_filter_not_installed.sh - pipe filter packages that are / are not installed for easy script piping
    • Alpine:
      • apk_install_packages.sh / apk_remove_packages.sh - installs Alpine apk package lists from arguments, files or stdin. Auto calls sudo if not root, accepts NO_FAIL=1 env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across Alpine versions)
      • apk_install_packages_if_absent.sh - installs Alpine apk packages only if not already installed, saving time and minimizing install logs / CI logs, plus all the features of apk_install_packages.sh above
      • apk_filter_installed.sh / apk_filter_not_installed.sh - pipe filter packages that are / are not installed for easy script piping
    • Mac:
      • brew_install_packages.sh / brew_remove_packages.sh - installs Mac Hombrew package lists from arguments, files or stdin. Accepts NO_FAIL=1 env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across versions)
      • brew_install_packages_if_absent.sh - installs Mac Homebrew packages only if not already installed, saving time and minimizing install logs / CI logs, plus all the features of brew_install_packages.sh above
      • brew_filter_installed.sh / brew_filter_not_installed.sh - pipe filter packages that are / are not installed for easy script piping
• all builds across all my GitHub repos now make system-packages before make pip / make cpan to shorten how many packages need installing, reducing chances of build failures

Builds, Languages & Linting

• check_*.sh - extensive collection of generalized tests - these run against all my GitHub repos via CI. Some examples:

  • Programming language linting:

    • Python (syntax, pep8, byte-compiling, reliance on asserts which can be disabled at runtime, except/pass etc.)
    • Perl
    • Java
    • Scala
    • Ruby
    • Bash / Shell
    • Misc (whitespace, custom code checks etc.)

  • Build System, Docker & CI linting:

    • Make
    • Maven
    • SBT
    • Gradle
    • Travis CI
    • Circle CI
    • GitLab CI
    • Concourse CI
    • Codefresh CI
    • Dockerfiles
    • Docker Compose
    • Vagrantfiles

Data Format Conversion & Validation

• csv_header_indices.sh - list CSV headers with their zero indexed numbers, useful reference when coding against column positions

• Data format validation validate_*.py from DevOps Python Tools repo:

  • CSV
  • JSON
  • Avro
  • Parquet
  • INI / Properties files (Java)
  • LDAP LDIF
  • XML
  • YAML

• json2yaml.sh - converts JSON to YAML

• yaml2json.sh - converts YAML to JSON - needed for some APIs like GitLab CI linting (see Gitlab section above)

See Also:

• DevOps Python Tools - 80+ DevOps CLI tools for AWS, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython

• SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery

• Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.

• Kubernetes templates - Kubernetes YAML templates - Best Practices, Tips & Tricks are baked right into the templates for future deployments

• The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)

• DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...

• HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.

• Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3

• Perl Lib - Perl utility library

• PyLib - Python utility library

• Lib-Java - Java utility library

• Nagios Plugin Kafka - Kafka Nagios Plugin written in Scala with Kerberos support

Pre-built Docker images are available for those repos (which include this one as a submodule) and the "docker available" icon above links to an uber image which contains all my github repos pre-built. There are Centos, Alpine, Debian and Ubuntu versions of this uber Docker image containing all repos.

Individual Setup Parts

Optional, only if you don't do the full make install.

Install only OS system package dependencies and AWS CLI via Python Pip (doesn't symlink anything to $HOME):

make

Adds sourcing to .bashrc and .bash_profile and symlinks dot config files to $HOME (doesn't install OS system package dependencies):

make link

undo via

make unlink

Install only OS system package dependencies (doesn't include AWS CLI or Python packages):

make system-packages

Install AWS CLI:

make aws

Install Azure CLI:

make azure

Install GCP GCloud SDK (includes CLI):

make gcp

Install GCP GCloud Shell environment (sets up persistent OS packages and all home directory configs):

make gcp-shell

Install generically useful Python CLI tools and modules (includes AWS CLI, autopep8 etc):

make python

Full Help

> make help

 Usage:

  Common Options:

    make help                   show this message
    make build                  installs all dependencies - OS packages and any language libraries via native tools eg. pip, cpanm, gem, go etc that are not available via OS packages
    make build-retry            retries 'make build' x 3 until success to try to mitigate temporary upstream repo failures triggering false alerts in CI systems
    make ci                     prints env, then runs 'build-retry' for more resilient CI builds with debugging
    make printenv               prints environment variables, CPU cores, OS release, $PWD, Git branch, hashref etc. Useful for CI debugging
    make system-packages        installs OS packages only (detects OS via whichever package manager is available)
    make test                   run tests
    make clean                  removes compiled / generated files, downloaded tarballs, temporary files etc.

    make submodules             initialize and update submodules to the right release (done automatically by build / system-packages)
    make init                   same as above, often useful to do in CI systems to get access to additional submodule provided targets such as 'make ci'

    make cpan                   install any modules listed in any cpan-requirements.txt files if not already installed

    make pip                    install any modules listed in any requirements.txt files if not already installed

    make python-compile         compile any python files found in the current directory and 1 level of subdirectory
    make pycompile

    make github                 open browser at github project
    make readme                 open browser at github's README
    make github-url             print github url and copy to clipboard
    make status                 open browser at Github CI Builds overview Status page for all projects

    make ls                     print list of code files in project
    make wc                     show counts of files and lines

  Repo specific options:

    make install                builds all script dependencies, installs AWS CLI, symlinks all config files to $HOME and adds sourcing of bash profile

    make link                   symlinks all config files to $HOME and adds sourcing of bash profile
    make unlink                 removes all symlinks pointing to this repo's config files and removes the sourcing lines from .bashrc and .bash_profile

    make python-desktop         installs all Python Pip packages for desktop workstation listed in setup/pip-packages-desktop.txt
    make perl-desktop           installs all Perl CPAN packages for desktop workstation listed in setup/cpan-packages-desktop.txt
    make ruby-desktop           installs all Ruby Gem packages for desktop workstation listed in setup/gem-packages-desktop.txt
    make golang-desktop         installs all Golang packages for desktop workstation listed in setup/go-packages-desktop.txt
    make nodejs-desktop         installs all NodeJS packages for desktop workstation listed in setup/npm-packages-desktop.txt

    make desktop                installs all of the above + many desktop OS packages listed in setup/

    make mac-desktop            all of the above + installs a bunch of major common workstation software packages like Ansible, Terraform, MiniKube, MiniShift, SDKman, Travis CI, CCMenu, Parquet tools etc.
    make linux-desktop

    make ls-scripts             print list of scripts in this project, ignoring code libraries in lib/ and .bash.d/

    make kubernetes             installs kubectl and kustomize to ~/bin/
    make vim                    installs Vundle and plugins
    make tmux                   installs TMUX TPM and plugin for kubernetes context
    make ccmenu                 installs and (re)configures CCMenu to watch this and all other major HariSekhon GitHub repos
    make status                 open the Github Status page of all my repos build statuses across all CI platforms

    make aws                    installs AWS CLI tools
    make azure                  installs Azure CLI
    make gcp                    installs Google Cloud SDK

    make gcp-shell              sets up GCP Cloud Shell: installs core packages and links configs
                                (maintains itself across future Cloud Shells via .customize_environment hook)
    make azure-shell            sets up Azure Cloud Shell (limited compared to gcp-shell, doesn't install OS packages since there is no sudo)

Now exiting usage help with status code 3 to explicitly prevent silent build failures from stray 'help' arguments
make: *** [help] Error 3

(make help exits with error code 3 like most of my programs to differentiate from build success to make sure a stray help argument doesn't cause silent build failure with exit code 0)

Stargazers over time

git.io/bash-tools