Monitor progress of cp, mv, gzip, mysql and CLI commands

To monitor progress of an ongoing copy or mysql dump import you might try progress CLI tool. Track progress of commands like cp, mv, dd, tar, cat, rsync, grep, fgrep, egrep, cut, sort, md5sum, sha1sum, sha224sum, sha256sum, sha384sum, sha512sum, adb, gzip, gunzip, bzip2, bunzip2, xz, unxz, lzma, unlzma, 7z, 7za, zcat, bzcat, lzcat, split, gpg, etc.

Advertisements

Hikvision NVR camera stream watching on LG TV

LG Smart World contains the Smart IPTV app
ss_2018-10-13-11-58-51

What’s needed is creating a m3u playlist file with the RTSP stream URLs

#EXTM3U
#EXTINF:0,group-title="Kamera",Hatso resz
rtsp://admin:PASSWORD@192.168.88.3:6911/Streaming/Channels/101
#EXTINF:0,group-title="Kamera",Utca
rtsp://admin:PASSWORD@192.168.88.3:6911/Streaming/Channels/201

The m3u file can be uploaded via https://siptv.app/mylist/. You could upload it on a shared web hosting or on your own public web server that can realtime provide updated m3u file.

Tried mitmproxy & mitmweb

Today I came by https://mitmproxy.org/ and gave it a shot just to know what it is capable of. For example for REST API communication debugging. The mitmweb is in beta, but looks good and also worked for me 🙂

The best is that they provide a docker image so trying it out was really simple.

  1. Just run docker run --rm -it -p 8080:8080 -p 8081:8081 --net="host" mitmproxy/mitmproxy mitmweb
  2. Open http://127.0.01:8081
  3. I changed Firefox to use proxy at 127.0.0.1 and port 8080 and browsed the web with mitmproxy intercepting the HTTP communication

mitmproxy & mitmweb

Tinkering with GNU parallel and wget for broken link checking

Finally found a parallel spidering solution. Online solutions didn’t really fit, because I don’t want to overload the production site and they can’t reach http://localhost. Trying out parallel + wget snippet from https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Breadth-first-parallel-web-crawler-mirrorer looks promising.

#!/bin/bash

URL=$1
 # Stay inside the start dir
 BASEURL=$(echo $URL | perl -pe 's:#.*::; s:(//.*/)[^/]*:$1:')
 URLLIST=$(mktemp urllist.XXXX)
 URLLIST2=$(mktemp urllist.XXXX)
 SEEN=$(mktemp seen.XXXX)

# Spider to get the URLs
 echo $URL >$URLLIST
 cp $URLLIST $SEEN

while [ -s $URLLIST ] ; do
 cat $URLLIST |
 parallel lynx -listonly -image_links -dump {} \; \
 wget -qm -l1 -Q1 {} \; echo Spidered: {} \>\&2 |
 perl -ne 's/#.*//; s/\s+\d+.\s(\S+)$/$1/ and
 do { $seen{$1}++ or print }' |
 grep -F $BASEURL |
 grep -v -x -F -f $SEEN | tee -a $SEEN > $URLLIST2
 mv $URLLIST2 $URLLIST
 done

rm -f $URLLIST $URLLIST2 $SEEN

Great exercise for the CPUs
htop gnu parallel

When the command finishes then the next step is parsing access_log

grep -r ' 404 ' /var/log/httpd/access_log | cut -d ' ' -f 7 | sed -r 's/^\//http\:\/\/localhost\//g'

First experience with RAID0

In my new desktop PC I have Intel Rapid Storage technology so gave it a try. Bought a second Kingston V300 SSD to put it in RAID0.

Intel Rapid Storage technology
Intel Rapid Storage technology

I was curious about the performance with and without RAID0 of the SSDs. So I run hdparm -tT on them.
The results:

Kingston SSD performance
Kingston SSD performance

Kingston SSD performance

The old SSD slowed down the RAID array so I sold it and installed OS on the new SSD, because it was 3x faster and I didn’t need the extra space with two SSDs in RAID.
Unfortunately I don’t remember the speed of the old SSD when I bouht it. Now I saved the new ones to be able to compare after some years.
It’s mindblowing how different/super fast is my PC with the new SSD.

Check out this awesome article about swapping https://rudd-o.com/linux-and-free-software/tales-from-responsivenessland-why-linux-feels-slow-and-how-to-fix-that

For disk benchmarking on Linux checkout https://wiki.archlinux.org/index.php/Benchmarking/Data_storage_devices

Browser memory usage

I was curious why did Chromium “eat” my system memory and so I checked chrome://chrome-urls/ to see what’s there for memory usage debugging.

Saw chrome://memory-internals/ and checked it out. There’s the proof for browser tab and extension usage.

ss_2016-01-31-06-40
Chrome memory usage data from chrome://memory-internals

Tried closing and reopening some tabs, but before that I typed free -h to check memory usage.

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        1.9G        1.3G         53M        682M        1.8G
Swap:          2.8G        276M        2.5G
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        1.9G        1.3G         53M        684M        1.9G
Swap:          2.8G        275M        2.5G
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        2.0G        1.2G         58M        694M        1.7G
Swap:          2.8G        273M        2.5

The numbers were not lying. So if you working with a lot of tabs and extensions more RAM will benefit your system.

Similiar goodies are available also in Firefox. At http://kb.mozillazine.org/About_protocol_links you can see a list of internal about:* sites of Firefox.

For memory usage visit about:memory and click Measure

ss_2016-01-31-06-57
Firefox memory usage

Enable MySQL file logging

Wanted to know what queries are executed so I had to enable it for MariaDB.

SET GLOBAL general_log = 'ON';
SET GLOBAL general_log_file = '/var/log/mysql.log';
SET GLOBAL slow_query_log = 'On';
SET GLOBAL slow_query_log_file = '/var/log/mysql-slow.log';

To see the changes:

SHOW VARIABLES LIKE 'general_log';
SHOW VARIABLES LIKE 'slow_query_log';