Collection of useful tricks and tips


Unix/Bash

Differences between two directories:

diff -qr dir1 dir2

List size of all directories:

du -sch .[!.]* * | sort -h

In-place editing of files (instead of: command file >tmp; mv tmp file)

command <(cat file) > file

Here strings <<<

command <<< "word"

Swap two variables using here strings (without temporary variable)

a=5; b=3; read a b <<<"$b $a"; echo $a $b

Bash Process Substitution <(command)

comm <(sort file1) <(sort file2)

More references:
[1] brace expansion http://tldp.org/LDP/abs/html/special-chars.html

desmos

Nice tricks: https://desmosgraphunofficial.wordpress.com/2019/02/11/undocumented-details (Length, list generator, fit, normaldist)

Expressions can be injected also via the browser console (from https://www.reddit.com/r/desmos/comments/l073ab/comment/gjs9syr/?utm_source=share&utm_medium=web2x&context=3) 
state = Calc.getState()
state.expressions.list.push({
    type: "expression",
    latex: "f\\left(x\\right)=x^{2}"
})
Calc.setState(state)
This can be further enhanced, to load a data file via a file button. Copy the code from https://gist.github.com/mzechmeister/7f7ce55dec843f4e6464403a74e71ecc into the browser console.

gnuplot

Plotting binary fits file (a simple one-dimensional spectrum) with gnuplot

pl "gj876_NIR_flux_oxt_2010-09-17.fits" binary format="%float" endian=big record=24739 skip=2880*22 us 0:1

Record is length of the spectrum (NAXIS1 from header). Skip is the size of the header which is always a multiple of 2880 (80x36). The multiple value is given by the number of header lines divided by 36.

Plots on different scales with axis x1y2

plot x, 10*x**2 axis x1y2

3d barchart of a function, also called lego plot in idl (gnuplot v4.4+ only, inspired by http://t16web.lanl.gov/Kawano/gnuplot/plotpm3d-e.html)

set samp 81 # sampling of special filename '++' for x (gives an oversampling of two)
set isosamples 81 # sampling of special filename '++' for y (gives an oversampling of two)
set ticslevel 0.
set hidden offset 0
splot [-20:20][-20:20] '++' using (ceil($1)-0.5) : (ceil($2)-0.5) : (exp( -(floor($2)-0.1*floor($1))**2 /15.)) w l t ''
# set terminal pngcairo size 480,360 font 'Arial,12' crop; set out '3Dbarchart.png'; replot;
3Dbarchart.png

For gnuplot v5.2, sampling can be specified on the plot line

splot sample [t=-20:20-0.5:0.5][u=-20:20-0.5:0.5] '++' using (ceil($1)-0.5) : (ceil($2)-0.5) : (exp( -(floor($2)-0.1*floor($1))**2 /15.)) w l t ''
Note the last index in sampling is always inclusive (even if irregular).



Plot all columns of one row of a file. This can be done with help of matrix

plot 'file.dat' matrix every ::5:10::10 us 1:3

Here we plot the column number against the "z"-values (us 1:3) choosing row number 11 (start_block=10 and end_block=10) and starting with column number 6 (start_point=5).

Plot the first derivative of data

plot y=NaN, "<seq 0 100 |awk '{print sin($1*0.1)}'", "" us 0:(dy=$1-y, y=$1, dy)

Higher derivative would be also possible.
http://www.vttoth.com/CMS/technical-notes/250-plotting-derivatives-in-gnuplot
http://stackoverflow.com/questions/15751226/how-can-i-plot-the-derivative-of-a-graph-in-gnuplot

Plot data with errors coded by transparency

plot [0:2*pi] '+' us 1:(s=rand(0)+0.1, sin($1)+s*invnorm(rand(0))):(s):((floor(255*(1-0.05/s))<<24)+0xFF) w e pt 7 ps 2 lc rgb variable, sin(x) lt 7 lw 2



awk

Create a histogram for positive real numbers:

awk '{counts[(int($1/size)+0.5)*size]++} END {for (word in counts) print word, counts[word]}' size=0.01 filename

Create a histogram for real numbers (needs a floor function):

awk '{x=$1/size; bin=int(x); if(x<0&&bin!=x) bin--; counts[(bin+0.5)*size]++} END {for (word in counts) print word, counts[word]}' size=0.01 filename

Remove duplicate and nonconsecutive lines using awk and associative array

awk '!($0 in array) { array[$0]; print }' filename

Reading fits header

awk --re-interval '$0=RT;/^END /{exit}' RS='.{80}' UVES.2002-12-07T07:04:42.159.fits

Reading fits header and split keywords, values, and comments into $1, $2, and $3, respectively:

awk --re-interval '$0=RT{print $0} /^END /{exit}' RS='.{80}' FS='=|/' UVES.2002-12-07T07:04:42.159.fits

http://www.catonmat.net/blog/ten-awk-tips-tricks-and-pitfalls

sed

Shortest matching (non-greedy, sed regex is greedy)

sed 's,"[^"]*",,g' <<< 'name="Peter" age="23"'

To grab only the matching regular expression use grep -o or awk

grep -o '"[^"]*"' <<< 'name="Peter" age="23"'

awk '!(NR%2)' RS='"' <<< 'name="Peter" age="23"'

which is very useful for xml and html files.

Reading fits header with sed

sed -n 'l 81; q' gj876_NIR_flux_oxt_2010-09-17.fits | sed 's/\\$//; /^END *$/q'
sed 's/.\{80\}/&\n/g; q' gj876_NIR_flux_oxt_2010-09-17.fits | sed '/^END *$/q'

vi

Set row number

:set nu

tar

This command will extract(-x) verbosely(-v) tar gz(-z) file(-f) to the desired location:

tar -xvzf filename.tar.gz -C /desired/path

wget

Get files from a web directory (-r recursive, -np no parent, -nH no host -A pattern):

wget -r -np --cut-dirs=4 -nH -A '*.txt' http://url/

rsync

Copy and synchronizing remotely (-v verbose, -a archive, -z compressed):

rsync -avz /home/src/*.fits user@institute.de:/desired/targetdir/

IDL

replicate a string array (which does not work with the rebin function; found here)

IDL> a = ['2008-07-25T20:56', '2008-07-25T21:32']
IDL> print, a[*,bytarr(10)]

showing the code of a routine (found here)

IDL> .run -t profiles

execute multiline statements (found here)

IDL> a = indgen(10)
IDL> .run
- for i=0,10 do begin
-     print, a[i]
-     print, 2*i+1
- endfor
- end

Working with fits

http://cerebus.as.arizona.edu/~ioannis/teaching/idl/tutorial_fits.html

html-latex

Embedded figures with latex formula via google chart api.
Enter this in web adres

https://chart.googleapis.com/chart?cht=tx&chl=E=mc^2

and you will get:

Python

Identifying bottlenecks using cProfile to check speed and performance (sorted by time)

python -m cProfile -s time test.py

advanced analysing

python -m cProfile -s time -o speed.txt test.py
and then in Python
import pstats
stats = pstats.Stats('speed.txt'); stats.sort_stats('time'); stats.reverse_order()
stats.print_stats()

debugging in iPython

[1] %run test.py

breakpoints in iPython (example at line number 30)

[1] run -d -b 30 test.py

Another way to create a dict

dict(red='rot', blue='blau', green='gruen')