Scraping
Download a web page an all its requisites:
1
$ wget -r -p -nc -nH --level=1 https://pypi.python.org/simple/python-ldap/Check the local SOCKS proxy started by a Tor Browser is working:
1
$ curl --preproxy 127.0.0.1:9150 "https://check.torproject.org"Reuse local Tor Browser proxy to download a video:
1
$ yt-dlp --proxy socks5://127.0.0.1:9150 "https://www.video-provider.com/watch/random_id"Create a PNG image of a rendered html page:
1
$ kwebdesktop 1024 768 capture.png https://slashdot.org/
Servers
Test that your site is sending gzipped content:
1
$ curl -i -H "Accept-Encoding: gzip,deflate" https://kevin.deldycke.com 2>&1 | grep gzipPing some pages on internet to force our corporate proxy to refresh its internal cache:
1
$ for EGG in BeautifulSoup PIL Plone; do wget --server-response -O /dev/null https://pypi.python.org/simple/$EGG/; doneDebug mysterious numbers (source):
1
$ echo 'obase=16; 1195725856' | bc | xxd -r -ps | od -cb20000000 G E T3107 105 124 04040000004
Certificates
Create a minimal self-signed unencrypted SSL certificate without issuer information and a validity period of 10 years:
1
$ openssl req -x509 -nodes -subj '/' -days 3650 -newkey rsa:2048 -keyout self-signed.pem -out self-signed.pemCreate a pair of SSL self-signed certificate and (unencrypted) private key (source):
1
$ openssl genrsa -out private.key 20482$ openssl req -new -subj '/' -key private.key -out certreq.csr3$ openssl x509 -req -days 3650 -in certreq.csr -signkey private.key -out self-signed.pem4$ rm certreq.csrView certificate details:
1
$ openssl x509 -noout -text -in self-signed.pemFetch from a website its first certificate of the chain:
1
$ openssl s_client -connect imap.gmail.com:993 -showcerts 2>&1 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | sed -ne '1,/-END CERTIFICATE-/p' > ~/gmail.pemFetch the certificate from a website (the one returned is the last of the chain):
1
$ openssl s_client -connect imap.gmail.com:993 -showcerts 2>&1 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | tac | sed -ne '1,/-BEGIN CERTIFICATE-/p' | tac > ./google.pem
MIME type
List all the different media types (formerly known as MIME types) of all the files in the www folder:
1
$ find ./www -type f -exec file --mime-type -b "{}" \; | sort | uniq
Markup
Search non-breakable spaces that doesnβt end with a semicolon:
1
$ grep -RIi --extended-regexp ' [^;]' ./