Scraping

  • Download a web page an all its requisites:

    1 $ wget -r -p -nc -nH --level=1 https://pypi.python.org/simple/python-ldap/
  • Check the local SOCKS proxy started by a Tor Browser is working:

    1 $ curl --preproxy 127.0.0.1:9150 "https://check.torproject.org"
  • Reuse local Tor Browser proxy to download a video:

    1 $ yt-dlp --proxy socks5://127.0.0.1:9150 "https://www.video-provider.com/watch/random_id"
  • Create a PNG image of a rendered html page:

    1 $ kwebdesktop 1024 768 capture.png https://slashdot.org/

Servers

  • Test that your site is sending gzipped content:

    1 $ curl -i -H "Accept-Encoding: gzip,deflate" https://kevin.deldycke.com 2>&1 | grep gzip
  • Ping some pages on internet to force our corporate proxy to refresh its internal cache:

    1 $ for EGG in BeautifulSoup PIL Plone; do wget --server-response -O /dev/null https://pypi.python.org/simple/$EGG/; done
  • Debug mysterious numbers (source):

    1 $ echo 'obase=16; 1195725856' | bc | xxd -r -ps | od -cb
    2 0000000   G   E   T
    3         107 105 124 040
    4 0000004

Certificates

  • Create a minimal self-signed unencrypted SSL certificate without issuer information and a validity period of 10 years:

    1 $ openssl req -x509 -nodes -subj '/' -days 3650 -newkey rsa:2048 -keyout self-signed.pem -out self-signed.pem
  • Create a pair of SSL self-signed certificate and (unencrypted) private key (source):

    1 $ openssl genrsa -out private.key 2048
    2 $ openssl req -new -subj '/' -key private.key -out certreq.csr
    3 $ openssl x509 -req -days 3650 -in certreq.csr -signkey private.key -out self-signed.pem
    4 $ rm certreq.csr
  • View certificate details:

    1 $ openssl x509 -noout -text -in self-signed.pem
  • Fetch from a website its first certificate of the chain:

    1 $ openssl s_client -connect imap.gmail.com:993 -showcerts 2>&1 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | sed -ne '1,/-END CERTIFICATE-/p' > ~/gmail.pem
  • Fetch the certificate from a website (the one returned is the last of the chain):

    1 $ openssl s_client -connect imap.gmail.com:993 -showcerts 2>&1 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | tac | sed -ne '1,/-BEGIN CERTIFICATE-/p' | tac > ./google.pem

MIME type

Markup

  • Search non-breakable spaces that doesn’t end with a semicolon:

    1 $ grep -RIi --extended-regexp '&nbsp[^;]' ./