Tried mitmproxy & mitmweb

Today I came by https://mitmproxy.org/ and gave it a shot just to know what it is capable of. For example for REST API communication debugging. The mitmweb is in beta, but looks good and also worked for me 🙂

The best is that they provide a docker image so trying it out was really simple.

  1. Just run docker run --rm -it -p 8080:8080 -p 8081:8081 --net="host" mitmproxy/mitmproxy mitmweb
  2. Open http://127.0.01:8081
  3. I changed Firefox to use proxy at 127.0.0.1 and port 8080 and browsed the web with mitmproxy intercepting the HTTP communication

mitmproxy & mitmweb

Advertisements

Tinkering with GNU parallel and wget for broken link checking

Finally found a parallel spidering solution. Online solutions didn’t really fit, because I don’t want to overload the production site and they can’t reach http://localhost. Trying out parallel + wget snippet from https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Breadth-first-parallel-web-crawler-mirrorer looks promising.

#!/bin/bash

URL=$1
 # Stay inside the start dir
 BASEURL=$(echo $URL | perl -pe 's:#.*::; s:(//.*/)[^/]*:$1:')
 URLLIST=$(mktemp urllist.XXXX)
 URLLIST2=$(mktemp urllist.XXXX)
 SEEN=$(mktemp seen.XXXX)

# Spider to get the URLs
 echo $URL >$URLLIST
 cp $URLLIST $SEEN

while [ -s $URLLIST ] ; do
 cat $URLLIST |
 parallel lynx -listonly -image_links -dump {} \; \
 wget -qm -l1 -Q1 {} \; echo Spidered: {} \>\&2 |
 perl -ne 's/#.*//; s/\s+\d+.\s(\S+)$/$1/ and
 do { $seen{$1}++ or print }' |
 grep -F $BASEURL |
 grep -v -x -F -f $SEEN | tee -a $SEEN > $URLLIST2
 mv $URLLIST2 $URLLIST
 done

rm -f $URLLIST $URLLIST2 $SEEN

Great exercise for the CPUs
htop gnu parallel

When the command finishes then the next step is parsing access_log

grep -r ' 404 ' /var/log/httpd/access_log | cut -d ' ' -f 7 | sed -r 's/^\//http\:\/\/localhost\//g'

First experience with RAID0

In my new desktop PC I have Intel Rapid Storage technology so gave it a try. Bought a second Kingston V300 SSD to put it in RAID0.

Intel Rapid Storage technology
Intel Rapid Storage technology

I was curious about the performance with and without RAID0 of the SSDs. So I run hdparm -tT on them.
The results:

Kingston SSD performance
Kingston SSD performance

Kingston SSD performance

The old SSD slowed down the RAID array so I sold it and installed OS on the new SSD, because it was 3x faster and I didn’t need the extra space with two SSDs in RAID.
Unfortunately I don’t remember the speed of the old SSD when I bouht it. Now I saved the new ones to be able to compare after some years.
It’s mindblowing how different/super fast is my PC with the new SSD.

Check out this awesome article about swapping https://rudd-o.com/linux-and-free-software/tales-from-responsivenessland-why-linux-feels-slow-and-how-to-fix-that

For disk benchmarking on Linux checkout https://wiki.archlinux.org/index.php/Benchmarking/Data_storage_devices

Browser memory usage

I was curious why did Chromium “eat” my system memory and so I checked chrome://chrome-urls/ to see what’s there for memory usage debugging.

Saw chrome://memory-internals/ and checked it out. There’s the proof for browser tab and extension usage.

ss_2016-01-31-06-40
Chrome memory usage data from chrome://memory-internals

Tried closing and reopening some tabs, but before that I typed free -h to check memory usage.

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        1.9G        1.3G         53M        682M        1.8G
Swap:          2.8G        276M        2.5G
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        1.9G        1.3G         53M        684M        1.9G
Swap:          2.8G        275M        2.5G
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        2.0G        1.2G         58M        694M        1.7G
Swap:          2.8G        273M        2.5

The numbers were not lying. So if you working with a lot of tabs and extensions more RAM will benefit your system.

Similiar goodies are available also in Firefox. At http://kb.mozillazine.org/About_protocol_links you can see a list of internal about:* sites of Firefox.

For memory usage visit about:memory and click Measure

ss_2016-01-31-06-57
Firefox memory usage

Enable MySQL file logging

Wanted to know what queries are executed so I had to enable it for MariaDB.

SET GLOBAL general_log = 'ON';
SET GLOBAL general_log_file = '/var/log/mysql.log';
SET GLOBAL slow_query_log = 'On';
SET GLOBAL slow_query_log_file = '/var/log/mysql-slow.log';

To see the changes:

SHOW VARIABLES LIKE 'general_log';
SHOW VARIABLES LIKE 'slow_query_log';

logrotate error: stat of /var/log/xferlog failed

In Arch Linux the logrotate service was failing:

user@host# systemctl start logrotate.service 
Job for logrotate.service failed. See "systemctl status logrotate.service" and "journalctl -xe" for details.

So I ran it by hand to debug:

user@host# logrotate /etc/logrotate.conf

And the following error appeared:

logrotate error: stat of /var/log/xferlog failed

I fired up a grep for xferlog in the /etc directory:

user@host:/etc# grep -r xferlog *
logrotate.d/proftpd:/var/log/xferlog

Solution was commenting the xferlog rule in logrotate.d/proftpd. I don’t need the transfer log.