Archive for the 'Blog' Category

Stata syntax highlighting in GEdit

Stata.lang file goes in ~/.local/share/gtksourceview-3.0/language-specs

Hundreds of datasets from libraries, archives, and museums

Douglas McCarthy and Andrea Wallace produced a survey of data access policies and data sources.  It’s in Google Docs and called the Survey of GLAM open access policy and practice [2.0]. It’s an excellent resource with the license terms and web locations of hundreds of datasets from libraries, archives, and museums.

image

CentOS 7 Repository for Owncloud Desktop Client

CentOS 7 Repository for OwnCloud desktop sync client is online. This application works well with Nextcloud.  Use (k)statusnotifieritem/appindicator extension and Topicons plus extensions in Gnome to ensure the icon works as it should.

The Url is https://robertknight.io/el7/robertknightio.repo

To add the repo, type the following command:

yum-config-manager –add-repo https://robertknight.io/el7/robertknightio.repo

Desktop Shortcuts for Kids 5.2.1

This is the VBscript for shortcuts on Windows 10 that open in Edge browser.

' Desktop Shortcuts for Kids, for Windows 10
' Robert Knight
' bobby@rdknight.net
' version 5 - July 2017
' ----------------------------------------------------------' 
Option Explicit

Dim wscript_shell_object
Dim shortcut_working_directory
Dim shortcut_save_location
Dim shortcut_file_object
Dim EXEC
Dim ICON
Dim DESCRIPTION

Set wscript_shell_object = CreateObject("WScript.Shell")
shortcut_working_directory ="C:\Windows"
shortcut_save_location = wscript_shell_object.SpecialFolders("Desktop")
Set shortcut_file_object = wscript_shell_object.CreateShortcut(shortcut_save_location & "\" & DESCRIPTION &

CreateDesktopShortcut "microsoft-edge:https://www.amazon.com/gp/video/storefront/ref=sv_atv_3?ie=UTF8&merchId=primevideokids", _
"https://www.amazon.com/favicon.ico", shortcut_working_directory, "Amazon Kid's Video", wscript_shell_object, shortcut_save_location, _
shortcut_file_object

CreateDesktopShortcut "microsoft-edge:http://www.abcmouse.com", _
"https://www.abcmouse.com/favicon.ico", shortcut_working_directory, "ABC Mouse", wscript_shell_object, shortcut_save_location, _
shortcut_file_object

CreateDesktopShortcut "microsoft-edge:http://www.nickjr.com", _
"http://www.nickjr.com/favicon.ico", shortcut_working_directory, "Nick Jr.", wscript_shell_object, shortcut_save_location, _
shortcut_file_object

CreateDesktopShortcut "microsoft-edge:http://www.cbc.ca/kidscbc2/", _
"http://www.cbc.ca/favicon.ico", shortcut_working_directory, "CBC 2", wscript_shell_object, shortcut_save_location, _
shortcut_file_object

CreateDesktopShortcut "microsoft-edge:http://www.sproutonline.com", _
"http://www.sproutonline.com/sites/sprout/themes/sprout_twig/images/favicon.ico", shortcut_working_directory, "Sprout", wscript_shell_object, shortcut_save_location, _
shortcut_file_object

CreateDesktopShortcut "microsoft-edge:http://www.youtube.com", _
"http://youtube.com/favicon.ico", shortcut_working_directory, "YouTube", wscript_shell_object, shortcut_save_location, _
shortcut_file_object

CreateDesktopShortcut "microsoft-edge:https://www.youtube.com/user/HobbyKidsTV", _
"http://icons.veryicon.com/ico/Internet%20%26%20Web/Black%20and%20White%20Social/youtube.ico", shortcut_working_directory, "Hobby Kids TV", wscript_shell_object, shortcut_save_location, _
shortcut_file_object

CreateDesktopShortcut "microsoft-edge:https://www.youtube.com/user/FGTeeV", _
"http://icons.veryicon.com/ico/Internet%20%26%20Web/Elegant%20Social%20Media/youtube.ico", shortcut_working_directory, "FGTeeV", wscript_shell_object, shortcut_save_location, _
shortcut_file_object

CreateDesktopShortcut "microsoft-edge:http://www.starfall.com/", _
"http://www.starfall.com/favicon.ico", shortcut_working_directory, "Starfall", wscript_shell_object, shortcut_save_location, _
shortcut_file_object

sub CreateDesktopShortcut(EXEC, ICON, shortcut_working_directory, DESCRIPTION, wscript_shell_object, shortcut_save_location, shortcut_file_object)
    Set shortcut_file_object = wscript_shell_object.CreateShortcut(shortcut_save_location & "\" & DESCRIPTION & ".lnk")
    shortcut_file_object.Description = DESCRIPTION
    shortcut_file_object.IconLocation = ICON
    shortcut_file_object.TargetPath = EXEC
    shortcut_file_object.WindowStyle = 3
    shortcut_file_object.WorkingDirectory = shortcut_working_directory
    shortcut_file_object.Save
End sub



WScript.Quit


Data to S3 with command line and encryption

This article details how to archive folders with their contents, encrypt the archives, upload them to S3, and remove the local copies.  A typical usage might be to archive the version of a website that I have with twenty thousand files on a certain date before changing the folder layouts of images for ten thousand images.

With about 345 GB of data in S3, I run into some specific situations that merit custom scripts to solve issues. One of these is that I like to create a snapshot of a folder at specific point in time before making adjustments to the scripts and software within. I also do this for documents and other folders so that a point-in-time backup exists. This way, if I decide I want a previous folder structure and content, I can just unzip the archive I created. The purpose of these backups is for a point-in-time previous version of a website or web-application in case I change my mind later on. They are rarely used. I need to copy a folder, and all of it’s contents recursively, and save the permissions, such as the execute permission on scripts. Then, to take that zip file and encrypt it using AES256 and upload it to an S3 bucket, and then delete the local zip file and encrypted file to free the disk space. The second component is a decryption script for when I download such a file and want to extract it.

There are some factors to consider, and one is storage costs. Encrypting the archives adds about 30% to the file size. S3 has an automatic transition to a lower cost storage tier available, so I set that on for the backup to use for the longer term storage.  The Standard Information Assurance tier is just over a penny per GB per month.  The second major consideration is security.  Amazon offers encryption of the S3 buckets and their contents using keys managed by Amazon. I turn this on.  My concern is hackers grabbing the data using stolen keys, and not avoiding the NSA or other government agencies.  If anyone will have Quantum computers, it will be the NSA, so if they want the data, they will get it.  With this in mind, I chose symmetric encryption and to use OpenSSL for the encryption.  GnuPG is a very popular and recommended application for this, and there are arguments against using OpenSSL for this purpose.  Ultimately it came down to the fact that the objective is using AES encryption with a password that can be passed in scripts rather than using public and private keys.  It may be a risk that Amazon’s encryption keys could be compromised allowing users to read the data. It is also likely that they have intrusion detection systems which would detect hackers attempting to brute force the passwords on the files within an S3 bucket.  It may also be that attackers downloaded all the files and attempt to brute force them locally.  I plan more details on this choice in a later blog post.

The choices are Zip file format so that the archives can be viewed on multiple platforms with ease, *nix file permission preservation on Linux, and AES encryption prior to upload, and decryption when the file is downloaded. To accomplish this, I created a folder in the home directory called Archives.  This Archives folder is where the zip files are created.  The next step is the creation of two scripts. I called these scrypts myencrypt and mydecrypt.  I store these in the home folder in a subdirectory called Scripts.  The next piece is two functions within the .bashrc file so that I can use the whole process at the command line.

myencrypt

#!/usr/bin/env bash

# $1 = input file
# $2 = output file
openssl enc -aes-256-cbc -salt -a -p -in $1 -out $1.encrypted -k “some-cool-password”

mydecrypt

#!/usr/bin/env bash

# $1 = input file
# $2 = output file
openssl enc -aes-256-cbc -a -d -p -in $1 -out $2 -k “some-cool-password”

bashrc, ltsarchive()

ltsarchive() { timestamp=$(date +”%Y-%m-%d-%H%M%p”) && read -p “Enter folder name: ” name && zip -rv9 ~/Archives/$name-$timestamp.zip $name && ~/Scripts/myencrypt ~/Archives/$name-$timestamp.zip && aws s3 cp ~/Archives/$name-$timestamp.zip.encrypted s3://bucket-name/long-term-archives/$name-$timestamp.zip.encrypted && rm -f ~/Archives/$name-$timestamp.zip && rm -f ~/Archives/$name-$timestamp.zip.encrypted; }

bashrc, ltcsdecrypt()

ltsdecrypt() { filepath=$(pwd) && read -p “Enter file name: ” name && newname=${name::-9} && ~/Scripts/mydecrypt $filepath/$name $filepath/$newname; }

For example, I would go into a directory and want to archive the Templates directory and all the contents.  I would type ltsarchive and then the shell would prompt me asking for the name of the folder.  I would then type Templates and the function runs and first zips the Templates folder and subcontents into ~/Archives/Templates-2018-07-23-2037PM.zip.  Then, those files are passed to OpenSSL which encrypts that file to Templates-2018-07-23-2037PM.zip.encrypted.  Then, that file uploads to the long term storage folder in the specified S3 bucket.  After that, the zip file and the .encrypted file are deleted from ~/Archives/.  Upon checking the S3 storage bucket, Templates-2018-07-23-2037PM.zip.encrypted appears.

Then, the files can be downloaded via the web browser or through the command line interface at a later time.  To decrypt Templates-2018-07-23-2037PM.zip.encrypted, we go into the directory at the shell and type ltsdecrypt.  It asks for the file name, so we type Templates-2018-07-23-2037PM.zip.encrypted.  It decryptes the file leaves Templates-2018-07-23-2037PM.zip in the same folder as the downloaded file.  These scripts can then be used in automation.

 

Try blocks in R for error handling

R scripts halt execution when an error is encountered.  When batch processing this produces situations where the error can not be merely skipped like it could in a .net application with  user involvement. To use Try() blocks for error handling in R, enclose the entire block of code in a try() block.

try({
# Do some stuff here
png(filename=args[3], width=900, height=300)
# some more code here...
dev.off()
})

Search Helper with Vivaldi

I just updated Search Helper with a Vivaldi browser selection.  It works on Gnome 3.

Android Studio desktop icon

Android Studio for Linux needs a desktop icon.  This is a functioning one to place in ~/.local/share/applications.

[Desktop Entry]
Version=1.0
Name=Android Studio
Comment=Software Development
GenericName=AndroidStudio
Exec=sh /opt/android-studio/bin/studio.sh
Terminal=false
X-MultipleArgs=false
Type=Application
Icon=/opt/android-studio/bin/studio.png
Categories=Development;
MimeType=text/html;text/xml;application/xhtml+xml
Keywords=programming;java;android;ide;

Name[en_US]=AndroidStudio

Yum update memory allocation issues

Yum Update Memory Allocation issues on 512M Virtual Machines:

Create a Swap file:

dd if=/dev/zero of=/swapfile bs=1024 count=655366
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

To disable and remove after updates are complete:
swapoff -a
rm -f /swapfile

STEM jobs in the United States

The number of science, technology, engineering, and math, STEM, jobs in the United States, shrank for the past three decades,1982-2012. The draw-down accelerated from 2000-2012.

The highest occupational growth occurred among occupations with soft skills, with K-12 teaching and non-doctor health care support staff, such as nurses, technicians, and therapists. From 2000-2012, those in the physical sciences, such as chemistry, physics, and others, biological scientists, and engineers saw decreases in the availability of work in their field. The percentage of the workforce that fell into the category of “engineer” declined by over 15% (David Deming, 2017). In “The Economics of Noncognitive Skills”, data from the Brookings Institution’s Hamilton project shows that the number of service jobs increased the most over the last three decades (Timothy Taylor, 14 October 2016). These are tasks such as customer service .