Merge branch 'master' into u/rtanaka/organizing

doc/OneStepSetup.md

@@ -92,44 +92,6 @@ At this point the next boot should start the Dashcam/music drives like normal. I
 
 > NOTE: Don't delete the `TESLAUSB_SETUP_FINISHED` or `WIFI_ENABLED` files. This is how the system knows setup is complete. 
 
-### Image builder source and patches
+# Image modification sources
 
-For now the image creation work is at:
-* Modified pi-gen [rtgoodwin's fork of pi-gen](https://github.com/rtgoodwin/pi-gen) in (whatever current branch I'm working at the time). 
-
-### Image refinement TODOs
-1. ~~Patch the hostname to teslausb~~
-1. Make it so if someone deletes the `TESLAUSB_SETUP_FINISHED` file it's handled gracefully. (Right now it will try to re-run setup which should be fine.) 
-1. Cache the remount packages? Might mess with first boot like `rsyslog`
-1. Aspirational TODO: Remove more packages and set services to stopped to make the boot process faster? 
-1. At this point, it's designed to pull the setup scripts _dynamically_, since development is still ongoing. If/when we reach a good frozen state, we can generate an image that is essentially ready to run. I think it'll also be pretty tricky to do some of the remounting and creating the backing files etc. on the image creation side. Open to suggestions/contributions there though! 
-
-
-#### Modifications to pi-gen builder from master
-
-The image is built on a Raspberry Pi 3B running Stretch, for maximum Pi-ception. 
-
-This is the basic configuration, but it's helpful to just [look at the code itself](https://github.com/rtgoodwin/pi-gen/tree/teslausb-headless-image) and the Readme for Pi-gen which explains this all in much greater detail:
-
-1. Added SKIP and SKIP_IMAGES files to stage3, 4, and 5 (if present). We want to build the default image up to stage2, then add our own stage for tweaks we want. 
-1. Added a `stage6` (or 7, just something beyond stage5). (There are stages 0-4 in the main Raspbian pi-gen repo by default, but may be a stage5 in some cases. This will help keep a clean merge later.)
-1. Copy the prerun.sh from `stage2`. Be SURE to `chmod +x` it. 
-1. Remove or rename the EXPORT_NOOBS files in all stages. We don't need a NOOBS image built. 
-1. In `stage6`, create a `00-tweaks` folder, with a `00-patches` folder and patches inside. Currently patched:
-
-    | File  | Change  |
-    |---|---|
-    | `cmdline.txt`| Add the dwc2,g_ether modules  |
-    | `config.txt`| Add the dwc2 module  |
-    | `hosts` | Change hostname to `teslausb` |
-    | `hostname` | Change hostname to `teslausb` |
-    
-    * The build process uses `quilt` for patching
-    * The path for any patching you do at this stage is `stage6/rootfs/FILEPATH` where `rootfs` represents the Pi's `/`. So, `cmdline.txt` is `stage6/rootfs/boot/cmdline.txt`.
-
-1. Added a file called `series` in the patches directory with the name of each `.diff` file in the order you want them applied.
-1. Added a `files` folder in stage6 with modified `rc.local`, and whatever else you want copied into the image. The modified `rc.local` will handle pulling down the `setup-teslausb-headless` file the first time and doing Wifi setup.
-1. Added a script to flash the Pi LED  
-1. Files are moved into final locations using the  `00-run.sh` script using the `install` command. See the script for details. I also `touch /boot/ssh` here so SSH is ready out of the box.
-1. Run `sudo ./build.sh` from the `pi-gen` directory.
-1. If you get a failure, it's almost certainly after stage2, so you can add SKIP files in all successful stages and rerun `sudo CLEAN=1 ./build.sh`
+The sources for the image modifications, and details, are in the [pi-gen-sources folder](https://github.com/cimryan/teslausb/pi-gen-sources). 

pi-gen-sources/00-packages

@@ -0,0 +1 @@
+dos2unix

pi-gen-sources/00-patches/01-cmdline-update.diff

@@ -0,0 +1,5 @@
+--- stage7/rootfs/boot/cmdline.txt
++++ stage7/rootfs/boot/cmdline.txt
+@@ -1 +1 @@
+-dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh
++dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=ROOTDEV rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait modules-load=dwc2,g_ether quiet

pi-gen-sources/00-patches/02-config-update.diff

@@ -0,0 +1,7 @@
+--- stage7/rootfs/boot/config.txt
++++ stage7/rootfs/boot/config.txt
+@@ -54,3 +54,4 @@
+
+ # Enable audio (loads snd_bcm2835)
+ dtparam=audio=on
++dtoverlay=dwc2

pi-gen-sources/00-patches/03-hosts-update.diff

@@ -0,0 +1,9 @@
+--- stage7/rootfs/etc/hosts
++++ stage7/rootfs/etc/hosts
+@@ -3,4 +3,5 @@
+ ff02::1		ip6-allnodes
+ ff02::2		ip6-allrouters
+ 
+-127.0.1.1	raspberrypi
++127.0.0.1	teslausb
++

pi-gen-sources/00-patches/04-hostname-update.diff

@@ -0,0 +1,5 @@
+--- stage7/rootfs/etc/hostname
++++ stage7/rootfs/etc/hostname
+@@ -1 +1 @@
+-raspberrypi
++teslausb

pi-gen-sources/00-patches/series

@@ -0,0 +1,4 @@
+01-cmdline-update.diff
+02-config-update.diff
+03-hosts-update.diff
+04-hostname-update.diff

pi-gen-sources/00-run.sh

@@ -0,0 +1,55 @@
+#!/bin/bash -e
+touch "${ROOTFS_DIR}/boot/ssh"
+install -m 755 files/rc.local		"${ROOTFS_DIR}/etc/"
+install -m 755 files/stage_flash    "${ROOTFS_DIR}/etc/"
+install -m 666 files/teslausb_setup_variables.conf.sample    "${ROOTFS_DIR}/boot/"
+install -m 666 files/wpa_supplicant.conf.sample    "${ROOTFS_DIR}/boot/"
+install -d "${ROOTFS_DIR}/root/bin"
+install -m 755 files/enable_wifi.sh "${ROOTFS_DIR}/root/bin"
+
+# Below here is the rest of the stage2 (builds the Stretch lite image)
+# run script commented out just to give guidance on things that can be done. 
+
+# install -m 755 files/teslausb_setup_scripts/bin/* "${ROOTFS_DIR}/root/bin/"
+# install -d "${ROOTFS_DIR}/root/bin/tmp"
+# install -m 755 files/teslausb_setup_scripts/tmp/* "${ROOTFS_DIR}/root/bin/tmp/"
+
+# on_chroot << EOF
+# systemctl disable hwclock.sh
+# systemctl disable nfs-common
+# systemctl disable rpcbind
+# systemctl disable ssh
+# systemctl enable regenerate_ssh_host_keys
+# EOF
+
+# if [ "${USE_QEMU}" = "1" ]; then
+# 	echo "enter QEMU mode"
+# 	install -m 644 files/90-qemu.rules "${ROOTFS_DIR}/etc/udev/rules.d/"
+# 	on_chroot << EOF
+# systemctl disable resize2fs_once
+# EOF
+# 	echo "leaving QEMU mode"
+# else
+# 	on_chroot << EOF
+# systemctl enable resize2fs_once
+# EOF
+# fi
+
+# on_chroot << \EOF
+# for GRP in input spi i2c gpio; do
+# 	groupadd -f -r "$GRP"
+# done
+# for GRP in adm dialout cdrom audio users sudo video games plugdev input gpio spi i2c netdev; do
+#   adduser pi $GRP
+# done
+# EOF
+
+# on_chroot << EOF
+# setupcon --force --save-only -v
+# EOF
+
+# on_chroot << EOF
+# usermod --pass='*' root
+# EOF
+
+# rm -f "${ROOTFS_DIR}/etc/ssh/"ssh_host_*_key*

pi-gen-sources/files/enable_wifi.sh

@@ -0,0 +1,54 @@
+#! /bin/bash -eu 
+# This is a standalone script to just manually trigger
+# the steps to enable wifi so you don't have to run rc.local
+#
+# This is probably a model to follow, similar to some of the 
+# reorganization work going on with the main scripts.
+
+HEADLESS_SETUP=false
+
+if [ -e "/boot/teslausb-headless-setup.log" ]
+then
+    HEADLESS_SETUP=true
+fi
+
+
+function setup_progress () {
+  if [ $HEADLESS_SETUP = "true" ]
+  then
+    SETUP_LOGFILE=/boot/teslausb-headless-setup.log
+    echo "$( date ) : $1" >> "$SETUP_LOGFILE"
+  fi
+    echo $1
+}
+
+function enable_wifi () {
+  setup_progress "Detecting whether to update wpa_supplicant.conf"
+  if [ ! -z ${SSID+x} ] && [ ! -z ${WIFIPASS+x} ]
+  then
+      if [ ! -e /boot/WIFI_ENABLED ]
+      then
+        if [ -e /root/bin/remountfs_rw ]
+        then
+          /root/bin/remountfs_rw
+        fi
+        setup_progress "Wifi variables specified, and no /boot/WIFI_ENABLED. Building wpa_supplicant.conf."
+        cp /boot/wpa_supplicant.conf.sample /boot/wpa_supplicant.conf
+        sed -i'.bak' -e "s/TEMPSSID/${SSID}/g" /boot/wpa_supplicant.conf
+        sed -i'.bak' -e "s/TEMPPASS/${WIFIPASS}/g" /boot/wpa_supplicant.conf
+        cp /boot/wpa_supplicant.conf /etc/wpa_supplicant/wpa_supplicant.conf
+        touch /boot/WIFI_ENABLED
+        setup_progress "Rebooting..."
+        reboot 
+      fi
+  else 
+    echo "You need to export your desired SSID and WIFI pass like:"
+    echo "  export SSID=your_ssid"
+    echo "  export WIFIPASS=your_wifi_pass"
+    echo ""
+    echo "Then re-run enable_wifi.sh"
+
+  fi
+}
+
+enable_wifi

pi-gen-sources/files/hosts

@@ -0,0 +1,6 @@
+127.0.0.1	localhost
+::1		localhost ip6-localhost ip6-loopback
+ff02::1		ip6-allnodes
+ff02::2		ip6-allrouters
+
+127.0.0.1	raspberrypi

pi-gen-sources/files/rc.local

@@ -0,0 +1,153 @@
+#!/bin/bash -e
+#
+# rc.local
+#
+# This script is executed at the end of each multiuser runlevel.
+# Make sure that the script will "exit 0" on success or any other
+# value on error.
+#
+# In order to enable or disable this script just change the execution
+# bits.
+#
+# By default this script does nothing.
+
+# Print the IP address
+_IP=$(hostname -I) || true
+if [ "$_IP" ]; then
+  printf "My IP address is %s\n" "$_IP"
+fi
+
+
+SETUP_LOGFILE=/boot/teslausb-headless-setup.log
+
+function setup_progress () {
+    echo "$( date ) : $1" >> "$SETUP_LOGFILE"
+    echo $1
+}
+
+function get_script () {
+  local local_path="$1"
+  local name="$2"
+  local remote_path="${3:-}"
+  
+  curl -o "$local_path/$name" https://raw.githubusercontent.com/"$REPO"/teslausb/"$BRANCH"/"$remote_path"/"$name"
+  # wget -O "$local_path/$name" https://raw.githubusercontent.com/"$REPO"/teslausb/"$BRANCH"/"$remote_path"/"$name"
+  chmod +x "$local_path/$name"
+}
+
+function enable_wifi () {
+  setup_progress "Detecting whether to update wpa_supplicant.conf"
+  if [[ ! -z $SSID ]] && [[ ! -z $WIFIPASS ]]
+  then
+      if [ ! -e /boot/WIFI_ENABLED ]
+      then
+        if [ -e /root/bin/remountfs_rw ]
+        then
+          /root/bin/remountfs_rw
+        fi
+        setup_progress "Wifi variables specified, and no /boot/WIFI_ENABLED. Building wpa_supplicant.conf."
+        cp /boot/wpa_supplicant.conf.sample /boot/wpa_supplicant.conf
+        sed -i'.bak' -e "s/TEMPSSID/${SSID}/g" /boot/wpa_supplicant.conf
+        sed -i'.bak' -e "s/TEMPPASS/${WIFIPASS}/g" /boot/wpa_supplicant.conf
+        cp /boot/wpa_supplicant.conf /etc/wpa_supplicant/wpa_supplicant.conf
+        touch /boot/WIFI_ENABLED
+        setup_progress "Rebooting..."
+        reboot 
+      fi
+  fi
+}
+
+
+if [ -e "/boot/teslausb_setup_variables.conf" ]
+  then
+    if [ -e /root/bin/remountfs_rw ]
+    then
+      /root/bin/remountfs_rw
+    fi
+    cp /boot/teslausb_setup_variables.conf /root/
+    dos2unix /root/teslausb_setup_variables.conf
+    source "/root/teslausb_setup_variables.conf"
+  fi
+
+# Check for headless setup
+if [ -z "${HEADLESS_SETUP+x}" ]
+then
+  HEADLESS_SETUP=false
+fi
+
+
+# check if we previously marked wifi as enabled
+enable_wifi
+# Good to start setup at this point
+
+# This begins the Headless Setup loop 
+# If the FINISHED file does not exist then we start setup. Otherwise passes on to normal loop
+if [ ! -e "/boot/TESLAUSB_SETUP_FINISHED" ] && [ $HEADLESS_SETUP = "true" ]
+then
+  if [ -e /root/bin/remountfs_rw ]
+  then
+    /root/bin/remountfs_rw
+  fi
+  touch "/boot/TESLAUSB_SETUP_STARTED"
+ 
+    # Grab the setup variables. Should still be there since setup isn't finished.
+    # This is a double check to cover various scenarios of mixed headless/not headless setup attempts
+    if [ -e "/boot/teslausb_setup_variables.conf" ]
+    then
+      cp /boot/teslausb_setup_variables.conf /root/
+      dos2unix /root/teslausb_setup_variables.conf
+      source "/root/teslausb_setup_variables.conf"
+    else
+      # No conf file found, can't complete setup
+      setup_progress "Setup appears not to have completed, but you didn't provide a teslausb_setup_variables.conf."
+    fi
+
+
+
+    # Make the bin dir if needed to grab the setup script into it and persist
+    if [ ! -d "/root/bin" ]
+    then
+      mkdir "/root/bin"
+    fi
+
+    if [ ! -e "/root/bin/setup-teslausb" ] 
+    then
+      REPO=${REPO:-cimryan}
+      BRANCH=${BRANCH:-master}
+      # Script doesn't exist, grab it. 
+      setup_progress "Grabbing main setup file."
+      get_script /root/bin setup-teslausb setup/pi
+    fi
+
+    setup_progress "Starting setup."
+
+    # Update the archiveserver value if needed in script on disk
+    sed -i'.bak' -e "s/TEMPARCHIVESERVER/$archiveserver/g" /etc/rc.local
+
+    # Start setup. This should take us all the way through to reboot
+    /root/bin/setup-teslausb
+
+    # reboot for good measure, also restarts the rc.local script
+    reboot
+fi
+
+LOGFILE=/tmp/rc.local.log
+
+function log () {
+  echo "$( date ) ${1}" >> "$LOGFILE"
+}
+if [ -e "/root/bin/archiveloop" ]
+then
+  log "Launching archival script..."
+  /root/bin/archiveloop TEMPARCHIVESERVER &
+  log "All done"
+else
+  echo "Setup doesn't seem to have completed, there is no /root/bin/archiveloop."
+  echo "Try re-running /root/bin/setup-teslausb (re-downloading if needed),"
+  echo "or export HEADLESS=true and run /etc/rc.local if you want to run automatic setup."
+fi
+exit 0
+
+
+
+

pi-gen-sources/files/stage_flash

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+i=$1
+flash_count=0
+flash_count=$((i + 1 )) 
+
+function led_off () {
+echo 1 | sudo tee /sys/class/leds/led0/brightness > /dev/null
+}
+
+function led_on () {
+echo 0 | sudo tee /sys/class/leds/led0/brightness > /dev/null
+}
+
+function led_flash () {
+   led_on
+   sleep .2
+   led_off
+}
+
+function flash () {
+  local COUNTER=0
+  sleep 2
+  while [ ${COUNTER} -lt ${flash_count} ]
+  do  
+    led_flash
+    let COUNTER=COUNTER+1
+  done
+}
+
+flash
+sleep 5
+
+exit 0

pi-gen-sources/files/teslausb_setup_variables.conf.sample

@@ -0,0 +1,43 @@
+#####################################################################
+# SAMPLE CONFIGURATION FILE FOR TESLAUSB Pi Setup
+#
+# Example config file for teslausb_setup. Lines with "#" are comments/ignored.
+# Remove the "#" before "export" to activate a line. Be sure to rename this file
+# to "teslausb_setup_variables.conf" and place it in the "boot" folder of your
+# SD card. 
+#
+######################################################################
+
+# Default variables for CIFS (Windows/Mac file sharing) setup
+export archiveserver=your_archive_name_or_ip
+export sharename=your_archive_share_name
+export shareuser=username
+export sharepassword=password
+export campercent=100
+
+# Wifi setup information
+export SSID=your_ssid
+export WIFIPASS=your_pass
+
+# If doing a headless (i.e. automatic) setup
+export HEADLESS_SETUP=true
+
+# Uncomment if setting up Pushover push notifications 
+# export pushover_enabled=false
+# export pushover_user_key=user_key
+# export pushover_app_key=app_key
+
+# Uncomment and change if you want setup scripts to be pulled
+# from a different repo than the "main" github.com/cimryan/teslausb
+# export REPO=cimryan
+
+# Uncomment and change if you want a different branch than master
+# export BRANCH=master
+
+# Uncomment if you're using rsync/ssh for the archive. 
+# Note: RSYNC_ENABLE=true is going to disable the default archive server.
+#
+# export RSYNC_ENABLE=true
+# export RSYNC_USER=<ftp username>
+# export RSYNC_SERVER=<ftp IP/host>
+# export RSYNC_PATH=<destination path to save in>

pi-gen-sources/files/wpa_supplicant.conf.sample

@@ -0,0 +1,8 @@
+ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
+update_config=1
+
+network={
+  ssid="TEMPSSID"
+  psk="TEMPPASS"
+  key_mgmt=WPA-PSK
+}