Merge pull request #78 from cimryan/u/rtgoodwin/headless-patch

U/rtgoodwin/headless patch

README.md

@@ -32,13 +32,13 @@ Required:
   * A USB A Add-on Board if you want to plug your Pi into your Tesla like a USB drive instead of using a cable. [Amazon](https://www.amazon.com/gp/product/B07BK2BR6C/)
 
 Optional:
-* A case. Don't want unprotected circuits hanging about! Official case at [Adafruit](https://www.adafruit.com/product/2885) or [Amazon](https://www.amazon.com/gp/product/B06Y593MHV). There are many others to choose from. Note that the official case won't work with the USB A Add on board.
+* A case. The "Official" case: [Adafruit](https://www.adafruit.com/product/2885) or [Amazon](https://www.amazon.com/gp/product/B06Y593MHV). There are many others to choose from. Note that the official case won't work with the USB A Add on board.
 * USB Splitter if you don't want to lose a front USB port. [The Onvian Splitter](https://www.amazon.com/gp/product/B01KX4TKH6) has been reported working by multiple people on reddit.
 
 ### Software
-Download [Raspbian Stretch Lite](https://www.raspberrypi.org/downloads/raspbian/)
+Download: [Raspbian Stretch Lite](https://www.raspberrypi.org/downloads/raspbian/)
-Download and install:
+
-* [Etcher](http://etcher.io)
+Download and install: [Etcher](http://etcher.io)
  
 ## Set up the Raspberry Pi
 There are four phases to setting up the Pi:
@@ -47,12 +47,20 @@ There are four phases to setting up the Pi:
 1. Set up the archive for dashcam clips.
 1. Set up the USB storage functionality.
 
+There is a streamlined process for setting up the Pi which can currently be used if you plan to use Windows file shares, MacOS Sharing, or Samba on Linux for your video archive. [Instructions](doc/OneStepSetup.md).
+
+If you'd like to host the archive using another technology or would like to set the Pi up, yourself, continue these instructions. 
+
 ### Get the OS onto the MicroSD card
 [These instructions](https://www.raspberrypi.org/documentation/installation/installing-images/README.md) tell you how to get Raspbian onto your MicroSD card. Basically:
 1. Connect your SD card to your computer.
 2. Use Etcher to write the zip file you downloaded to the SD card.
    > Note: you don't need to uncompress the zip file you downloaded.
 
+#### (Experimental) 
+
+There is an experimental image that requires only a configuration file. Needs testing and issues filed _on this specific image/process_ at [this repo](https://github.com/rtgoodwin/teslausb/tree/headless-patch/headless-scripts). Issues about general functionality should still be filed on the main (cimryan) repo. 
+
 ### Get a shell on the Pi
 Follow the instructions corresponding to the OS you used to flash the OS onto the MicroSD card:
 * Windows: [Instructions](doc/GetShellWithoutMonitorOnWindows.md).

doc/OneStepSetup.md

@@ -0,0 +1,135 @@
+# One-step setup
+
+This is a streamlined process for setting up the Pi. You'll flash a preconfigured version of Raspbian Stretch Lite and then fill out a config file. 
+
+## Notes 
+
+* Assumes your Pi has access to Wifi, with internet access (during setup). (But all setup methods do currently.) USB networking is still enabled for troubleshooting or manual setup
+* This image will work for either _headless_ (tested) or _manual_ (tested less) setup.
+* Currently not tested with the RSYNC/SFTP method when using headless setup. 
+
+## Configure the SD card before first boot of the Pi
+
+1. Flash the [latest image release](https://github.com/rtgoodwin/teslausb/releases) using Etcher or similar. 
+
+### For headless (automatic) setup
+
+1. Mount the card again, and in the `boot` directory create a `teslausb_setup_variables.conf` file to export the same environment varibles normally needed for manual setup (including archive info, Wifi, and push notifications (if desired). 
+A sample conf file is located in the `boot` folder on the SD card. 
+
+
+    The file should contain the entries below at a minimum, but **replace with your own values**:
+    ```
+    export archiveserver=Nautilus
+    export sharename=SailfishCam
+    export shareuser=sailfish
+    export sharepassword=pa$$w0rd
+    export campercent=100
+    export SSID=your_ssid
+    export WIFIPASS=your_wifi_password
+    export HEADLESS_SETUP=true
+    # export REPO=rtgoodwin
+    # export BRANCH=headless-patch
+    # By default will use the main repo, but if you've been asked to test the image, 
+    # these variables should be uncommunted and updated to point to the right repo/branch 
+
+    # export pushover_enabled=false
+    # export pushover_user_key=user_key
+    # export pushover_app_key=app_key
+    ```
+
+* Boot it in your Pi, give it a bit, watching for a series of flashes (2, 3, 4, 5) and then a reboot and/or the CAM/music drives to become available on your PC/Mac. The LED flash stages are:
+
+| Stage (number of flashes)  |  Activity |
+|---|---|
+| 2 | Verify setup variables  |
+| 3 | Grab scripts to start/continue setup |
+| 4 | Create partition and files to store camera clips/music) |
+| 5 | Setup completed; remounting filesystems as read-only and rebooting |
+
+
+
+* The Pi should be available for `ssh` at `pi@teslausb.local`, over Wifi (if automatic setup works) or USB networking (if it doesn't). It takes about 5 minutes, or more depending on network speed, etc. 
+* If plugged into just a power source, or your car, give it a few minutes until the LED starts pulsing steadily which means the archive loop is running and you're good to go. 
+* You should see in `/boot` the `TESLAUSB_SETUP_FINISHED` and `WIFI_ENABLED` files as markers of headless setup success as well.
+
+### For manual setup
+
+1. After flashing the image, boot it in your Pi and:
+    *  connect via USB networking at `ssh pi@teslausb.local`. (The Pi must be connected to your PC and plugged into the port labeled USB on the Pi. Or...
+    * You can also just automate the Wifi portion of setup by creating the `boot/teslausb_setup_variables.conf` file and populating it with the `SSID` and `WIFIPASS` variables:
+    ```
+    export SSID=your_ssid
+    export WIFIPASS=your_wifi_pass
+    ```
+
+1. Once you have an `ssh` session, follow the steps starting at [Set up the USB storage functionality](https://github.com/cimryan/teslausb#set-up-the-usb-storage-functionality) in the main guide. 
+
+### Troubleshooting
+
+#### Headless (full or Wifi) setup 
+* `ssh` to `pi@teslausb.local` (assuming Wifi came up, or your Pi is connected to your computer via USB) and look at the `/boot/teslausb-headless-setup.log`. 
+* Try `sudo -i` and then run `/etc/rc.local`. The scripts are  fairly resilient to restarting and not re-running previous steps, and will tell you about progress/failure.
+* If Wifi didn't come up:
+    * Double-check the SSID and WIFIPASS variables in `teslausb_setup_variables.conf`, and remove `/boot/WIFI_ENABLED`, then booting the SD in your Pi to retry automatic Wifi setup. 
+  * If still no go, re-run `/etc/rc.local`
+  * If all else fails, copy `/boot/wpa_supplicant.conf.sample` to `/boot/wpa_supplicant.conf` and edit out the `TEMP` variables to your desired settings. 
+* (Note: if you get an error about `read-only filesystem`, you may have to `sudo -i` and run `/root/bin/remountfs_rw`.
+
+
+# Background information
+## What happens under the covers
+
+When the Pi boots the first time: 
+* A `/boot/teslausb-headless-setup.log` file will be created and stages logged. 
+* Marker files will be created in `boot` like `TESLA_USB_SETUP_STARTED` and `TESLA_USB_SETUP_FINISHED` to track progress. 
+* Wifi is detected by looking for `/boot/WIFI_ENABLED` and if not, creates the `wpa_supplicant.conf` file in place, using `SSID` and `WIFIPASS` from `teslausb_setup_varibles.conf` and reboots. 
+* The Pi LED will flash patterns (2, 3, 4, 5) as it gets to each stage (labeled in the setup-teslausb-headless script). 
+  * ~~10 flashes means setup failed!~~ (not currently working)
+* After the final stage and reboot the LED will go back to normal. Remember, the step to remount the filesystem takes a few minutes.
+
+At this point the next boot should start the Dashcam/music drives like normal. If you're watching the LED it will start flashing every 1 second, which is the archive loop running. 
+
+> NOTE: Don't delete the `TESLAUSB_SETUP_FINISHED` or `WIFI_ENABLED` files. This is how the system knows setup is complete. 
+
+### Image builder source and patches
+
+For now the image creation work is at:
+* Modified pi-gen [rtgoodwin's fork of pi-gen](https://github.com/rtgoodwin/pi-gen) in (whatever current branch I'm working at the time). 
+
+### Image refinement TODOs
+1. ~~Patch the hostname to teslausb~~
+1. Make it so if someone deletes the `TESLAUSB_SETUP_FINISHED` file it's handled gracefully. (Right now it will try to re-run setup which should be fine.) 
+1. Cache the remount packages? Might mess with first boot like `rsyslog`
+1. Aspirational TODO: Remove more packages and set services to stopped to make the boot process faster? 
+1. At this point, it's designed to pull the setup scripts _dynamically_, since development is still ongoing. If/when we reach a good frozen state, we can generate an image that is essentially ready to run. I think it'll also be pretty tricky to do some of the remounting and creating the backing files etc. on the image creation side. Open to suggestions/contributions there though! 
+
+
+#### Modifications to pi-gen builder from master
+
+The image is built on a Raspberry Pi 3B running Stretch, for maximum Pi-ception. 
+
+This is the basic configuration, but it's helpful to just [look at the code itself](https://github.com/rtgoodwin/pi-gen/tree/teslausb-headless-image) and the Readme for Pi-gen which explains this all in much greater detail:
+
+1. Added SKIP and SKIP_IMAGES files to stage3, 4, and 5 (if present). We want to build the default image up to stage2, then add our own stage for tweaks we want. 
+1. Added a `stage6` (or 7, just something beyond stage5). (There are stages 0-4 in the main Raspbian pi-gen repo by default, but may be a stage5 in some cases. This will help keep a clean merge later.)
+1. Copy the prerun.sh from `stage2`. Be SURE to `chmod +x` it. 
+1. Remove or rename the EXPORT_NOOBS files in all stages. We don't need a NOOBS image built. 
+1. In `stage6`, create a `00-tweaks` folder, with a `00-patches` folder and patches inside. Currently patched:
+
+    | File  | Change  |
+    |---|---|
+    | `cmdline.txt`| Add the dwc2,g_ether modules  |
+    | `config.txt`| Add the dwc2 module  |
+    | `hosts` | Change hostname to `teslausb` |
+    | `hostname` | Change hostname to `teslausb` |
+    
+    * The build process uses `quilt` for patching
+    * The path for any patching you do at this stage is `stage6/rootfs/FILEPATH` where `rootfs` represents the Pi's `/`. So, `cmdline.txt` is `stage6/rootfs/boot/cmdline.txt`.
+
+1. Added a file called `series` in the patches directory with the name of each `.diff` file in the order you want them applied.
+1. Added a `files` folder in stage6 with modified `rc.local`, and whatever else you want copied into the image. The modified `rc.local` will handle pulling down the `setup-teslausb-headless` file the first time and doing Wifi setup.
+1. Added a script to flash the Pi LED  
+1. Files are moved into final locations using the  `00-run.sh` script using the `install` command. See the script for details. I also `touch /boot/ssh` here so SSH is ready out of the box.
+1. Run `sudo ./build.sh` from the `pi-gen` directory.
+1. If you get a failure, it's almost certainly after stage2, so you can add SKIP files in all successful stages and rerun `sudo CLEAN=1 ./build.sh`

doc/teslausb_setup_variables.conf.sample

@@ -0,0 +1,43 @@
+#####################################################################
+# SAMPLE CONFIGURATION FILE FOR TESLAUSB Pi Setup
+#
+# Example config file for teslausb_setup. Lines with "#" are comments/ignored.
+# Remove the "#" before "export" to activate a line. Be sure to rename this file
+# to "teslausb_setup_variables.conf" and place it in the "boot" folder of your
+# SD card. 
+#
+######################################################################
+
+# Default variables for CIFS (Windows/Mac file sharing) setup
+export archiveserver=your_archive_name_or_ip
+export sharename=your_archive_share_name
+export shareuser=username
+export sharepassword=password
+export campercent=100
+
+# Wifi setup information
+export SSID=your_ssid
+export WIFIPASS=your_pass
+
+# If doing a headless (i.e. automatic) setup
+export HEADLESS_SETUP=true
+
+# Uncomment if setting up Pushover push notifications 
+# export pushover_enabled=false
+# export pushover_user_key=user_key
+# export pushover_app_key=app_key
+
+# Uncomment and change if you want setup scripts to be pulled
+# from a different repo than the "main" github.com/cimryan/teslausb
+# export REPO=cimryan
+
+# Uncomment and change if you want a different branch than master
+# export BRANCH=master
+
+# Uncomment if you're using rsync/ssh for the archive. 
+# Note: RSYNC is not usable for headless setup since it currently requires a manual step. 
+# Note: RSYNC_ENABLE=true is going to disable the default archive server.
+# export RSYNC_ENABLE=true
+# export RSYNC_USER=<ftp username>
+# export RSYNC_SERVER=<ftp IP/host>
+# export RSYNC_PATH=<destination path to save in>

run/cifs_archive/archive-clips.sh

@@ -19,6 +19,9 @@ for file_name in "$CAM_MOUNT"/TeslaCam/saved*; do
 done
 log "Moved $NUM_FILES_MOVED file(s)."
 
+if [ $NUM_FILES_MOVED -gt 0 ]
+then
 /root/bin/send-pushover "$NUM_FILES_MOVED"
+fi
 
 log "Finished moving clips to archive."

setup/pi/setup-teslausb

@@ -1,22 +1,91 @@
 #!/bin/bash -eu
 
-USER_ENABLED_PUSHOVER=false
+USER_ENABLED_PUSHOVER=${USER_ENABLED_PUSHOVER:-false}
-
+SETUP_LOGFILE=/boot/teslausb-headless-setup.log
 REPO=${REPO:-cimryan}
-
 BRANCH=${BRANCH:-master}
+HEADLESS_SETUP=${HEADLESS_SETUP:-false}
+USE_LED_FOR_SETUP_PROGRESS=true
+
+
 
 if ! [ $(id -u) = 0 ]
 then
-  echo "STOP: Run sudo -i."
+  setup_progress "STOP: Run sudo -i."
   exit 1
 fi
 
+function setup_progress () {
+  if [ $HEADLESS_SETUP = "true" ]
+  then
+    echo "$( date ) : $1" >> "$SETUP_LOGFILE"
+  fi
+    echo $1
+}
+
+function headless_setup_populate_variables () {
+  # Pull in the conf file variables to make avail to this script and subscripts
+  if [ -e /boot/teslausb_setup_variables.conf ] && [ $HEADLESS_SETUP = "true" ]
+  then
+    source /boot/teslausb_setup_variables.conf
+  fi
+}
+
+
+function headless_setup_mark_setup_failed () {
+  if [ $HEADLESS_SETUP = "true" ]
+  then
+    setup_progress "ERROR: Setup Failed."
+    touch /boot/TESLAUSB_SETUP_FAILED
+  fi
+}
+
+function headless_setup_mark_setup_success () {
+  if [ $HEADLESS_SETUP = "true" ]
+  then
+
+    if [ -e /boot/TESLAUSB_SETUP_FAILED ]
+    then
+      rm /boot/TESLAUSB_SETUP_FAILED
+    fi
+
+    rm /boot/TESLAUSB_SETUP_STARTED
+    touch /boot/TESLAUSB_SETUP_FINISHED
+    # This sed shouldn't be needed, but double checking just to be sure. 
+    sed -i'.bak' -e "s/TEMPARCHIVESERVER/$archiveserver/g" /etc/rc.local
+    setup_progress "Main setup completed. Remounting file systems read only."
+  fi
+}
+
+function headless_setup_progress_flash () {
+  if [ $USE_LED_FOR_SETUP_PROGRESS = "true" ] && [ $HEADLESS_SETUP = "true" ]
+  then
+    /etc/stage_flash $1
+  fi
+}
+
+function setup_led_off () {
+
+  if [ $USE_LED_FOR_SETUP_PROGRESS = "true" ] && [ $HEADLESS_SETUP = "true" ]
+  then
+    echo "none" | sudo tee /sys/class/leds/led0/trigger > /dev/null
+    echo 1 | sudo tee /sys/class/leds/led0/brightness > /dev/null
+  fi
+}
+
+function setup_led_on () {
+
+  if [ $USE_LED_FOR_SETUP_PROGRESS = "true" ] && [ $HEADLESS_SETUP = "true" ]
+  then
+    echo 0 | sudo tee /sys/class/leds/led0/brightness > /dev/null
+  fi
+}
+
 function check_variable () {
   local var_name="$1"
   if [ -z "${!var_name+x}" ]
   then
-    echo "STOP: Define the variable $var_name like this: export $var_name=value"
+    setup_progress "STOP: Define the variable $var_name like this: export $var_name=value"
     exit 1
   fi
 }
@@ -26,14 +95,14 @@ function check_pushover_enabled () {
   then
     if [ ! -n "${pushover_user_key+x}" ] || [ ! -n "${pushover_app_key+x}"  ]
     then
-      echo "STOP: You're trying to setup Pushover but didn't provide your User and/or App key."
+      setup_progress "STOP: You're trying to setup Pushover but didn't provide your User and/or App key."
-      echo "Define the variables like this:"
+      setup_progress "Define the variables like this:"
-      echo "export pushover_user_key=put_your_userkey_here"
+      setup_progress "export pushover_user_key=put_your_userkey_here"
-      echo "export pushover_app_key=put_your_appkey_here"
+      setup_progress "export pushover_app_key=put_your_appkey_here"
       exit 1
     elif [ "${pushover_user_key}" = "put_your_userkey_here" ] || [  "${pushover_app_key}" = "put_your_appkey_here" ]
     then
-      echo "STOP: You're trying to setup Pushover, but didn't replace the default User and App key values."
+      setup_progress "STOP: You're trying to setup Pushover, but didn't replace the default User and App key values."
       exit 1
     else
       USER_ENABLED_PUSHOVER=true
@@ -45,17 +114,17 @@ function check_pushover_enabled () {
 }
 
 function check_available_space () {
-  echo "Verifying that there is sufficient space available on the MicroSD card..."
+  setup_progress "Verifying that there is sufficient space available on the MicroSD card..."
 
   local available_space="$( parted -m /dev/mmcblk0 u b print free | tail -1 | cut -d ":" -f 4 | sed 's/B//g' )"
 
   if [ "$available_space" -lt  4294967296 ]
   then
-    echo "STOP: The MicroSD card is too small."
+    setup_progress "STOP: The MicroSD card is too small."
     exit 1
   fi
 
-  echo "There is sufficient space available."
+  setup_progress "There is sufficient space available."
 }
 
 function get_script () {
@@ -63,8 +132,10 @@ function get_script () {
   local name="$2"
   local remote_path="${3:-}"
   
-  wget -O "$local_path/$name" https://raw.githubusercontent.com/"$REPO"/teslausb/"$BRANCH"/"$remote_path"/"$name"
+  curl -o "$local_path/$name" https://raw.githubusercontent.com/"$REPO"/teslausb/"$BRANCH"/"$remote_path"/"$name"
+  # wget -O "$local_path/$name" https://raw.githubusercontent.com/"$REPO"/teslausb/"$BRANCH"/"$remote_path"/"$name"
   chmod +x "$local_path/$name"
+  setup_progress "Downloaded $local_path/$name ..."
 }
 
 function get_ancillary_setup_scripts () {
@@ -75,11 +146,11 @@ function get_ancillary_setup_scripts () {
 
 function fix_cmdline_txt_modules_load ()
 {
-  echo "Fixing the modules-load parameter in /boot/cmdline.txt..."
+  setup_progress "Fixing the modules-load parameter in /boot/cmdline.txt..."
   cp /boot/cmdline.txt ~
   cat ~/cmdline.txt | sed 's/ modules-load=dwc2,g_ether/ modules-load=dwc2/' > /boot/cmdline.txt
   rm ~/cmdline.txt
-  echo "Fixed cmdline.txt."
+  setup_progress "Fixed cmdline.txt."
 }
 
 BACKINGFILES_MOUNTPOINT=/backingfiles
@@ -103,19 +174,20 @@ function create_usb_drive_backing_files () {
   
   if ! findmnt --mountpoint $BACKINGFILES_MOUNTPOINT
   then
-    echo "Mounting the partition for the backing files..."
+    setup_progress "Mounting the partition for the backing files..."
     mount $BACKINGFILES_MOUNTPOINT
-    echo "Mounted the partition for the backing files."
+    setup_progress "Mounted the partition for the backing files."
   fi
 
   if [ ! -e $BACKINGFILES_MOUNTPOINT/*.bin ]
   then
+    setup_progress "Creating backing disk files."
     /tmp/create-backingfiles.sh "$campercent" "$BACKINGFILES_MOUNTPOINT"
   fi
 }
 
 function configure_archive_scripts () {
-  echo "Configuring the archive scripts..."
+  setup_progress "Configuring the archive scripts..."
 
   get_script /root/bin archiveloop run
 
@@ -137,7 +209,7 @@ function configure_archive_scripts () {
 
   get_script /root/bin remountfs_rw run
   
-  echo "Configured the archive scripts."
+  setup_progress "Configured the archive scripts."
 }
 
 function configure_pushover_scripts() {
@@ -150,7 +222,7 @@ function configure_rc_local () {
     return
   fi
   
-  echo "Configuring /etc/rc.local to run the archive scripts at startup..."
+  setup_progress "Configuring /etc/rc.local to run the archive scripts at startup..."
   echo "#!/bin/bash -eu" > ~/rc.local
   echo "archiveserver=\"${archiveserver}\"" >> ~/rc.local
 cat << 'EOF' >> ~/rc.local
@@ -169,11 +241,14 @@ EOF
 
   cat ~/rc.local > /etc/rc.local
   rm ~/rc.local
-  echo "Configured rc.local."
+  setup_progress "Configured rc.local."
 }
 
 function configure_hostname () {
-  echo "Configuring the hostname..."
+  # Headless image already has hostname set
+  if [ ! $HEADLESS_SETUP = "true" ]
+  then
+    setup_progress "Configuring the hostname..."
 
     local new_host_name="teslausb"
     cp /etc/hosts ~
@@ -181,14 +256,23 @@ function configure_hostname () {
 
     cp /etc/hostname ~
     sed "s/raspberrypi/$new_host_name/g" ~/hostname > /etc/hostname
-  echo "Configured the hostname."
+    setup_progress "Configured the hostname."
+  fi
 }
 
 function make_root_fs_readonly () {
   /tmp/make-root-fs-readonly.sh
 }
 
-echo "Verifying environment variables..."
+headless_setup_populate_variables
+
+# If USE_LED_FOR_SETUP_PROGRESS = true. 
+setup_led_off
+
+# Flash for stage 2 headless (verify variables)
+headless_setup_progress_flash 1
+
+setup_progress "Verifying environment variables..."
 
 RSYNC_ENABLE="${RSYNC_ENABLE:-false}"
 RCLONE_ENABLE="${RCLONE_ENABLE:-false}"
@@ -218,6 +302,11 @@ check_variable "campercent"
 
 check_pushover_enabled
 
+# Flash for Stage 3 headless (grab scripts)
+headless_setup_progress_flash 2
+
+setup_progress "Downloading additional setup scripts."
+
 if [ ! -e /root/bin ]
 then
   mkdir /root/bin
@@ -247,6 +336,7 @@ get_ancillary_setup_scripts
 
 pushd ~
 
+
 configure_archive_scripts
 
 configure_pushover_scripts
@@ -255,6 +345,9 @@ fix_cmdline_txt_modules_load
 
 echo "" >> /etc/fstab
 
+# Flash for stage 4 headless (Create backing files)
+headless_setup_progress_flash 3
+
 create_usb_drive_backing_files
 
 /root/bin/configure-archive.sh
@@ -263,6 +356,14 @@ configure_rc_local
 
 configure_hostname
 
+# Flash for stage 5 headless (Mark success, FS readonly)
+headless_setup_progress_flash 4
+
+headless_setup_mark_setup_success
+
 make_root_fs_readonly
 
-echo "All done."
+# If USE_LED_FOR_SETUP_PROGRESS = true. 
+setup_led_on
+
+setup_progress "All done."