Merge pull request #42 from Gocnak/master

Archive to FTP server using rsync

README.md

@@ -55,7 +55,10 @@ Set up a share on a Windows (or macOS using Sharing, or Linux using Samba) machi
 
 Get the IP address of the archive machine. You'll need this later, so write it down, somewhere. You can do this by opening a command prompt on the archive machine and typing ipconfig. Get the IP address from the line labeled "IPv4 Address". These instructions will assume that the IP address of the archive server is 192.168.0.41.
 
-### TODO Other hosting solutions
+### Hosting via SFTP/rsync
+Since sftp/rsync is accessing a computer through SSH, the only requirement for hosting an SFTP/rsync server is to have a box running Linux. An example can be another Raspberry Pi connected to your local network with a USB storage drive plugged in. The official Raspberry Pi site has a good example on [how to mount an external drive](https://www.raspberrypi.org/documentation/configuration/external-storage.md). You will need the username and host/IP of the storage server, as well as the path for the files to go in, and the storage server will need to allow SSH.
+
+### ***TODO: Other hosting solutions***
 
 ## Set up the Raspberry Pi
 There are four phases to setting up the Pi:
@@ -89,13 +92,18 @@ Now that you have Wifi up and running, it's time to set up the USB storage and s
     ping 192.168.0.41
     ```
 1. If you can't ping the archive server by IP address from the Pi, you should go do whatever you need to on your network to fix that. If you can't reach the archive server by name, from the Pi but you can by IP address, then use its IP address, below, in place of its name.
-1. Run these commands, subsituting your values. The last line is the percent of the drive you want to allocate for dashcam storage. The remaining percentage will be allocated for music.
+1. Determine how much, as a percentage, of the drive you want to allocate to recording dashcam footage by using:
+    ```
+    export campercent=<number>
+    ```
+    For example, using `export campercent=100` would allocate 100% of the space to recording footage from your car, and would not create a separate music partition. `export campercent=50` would be only allocate half of the space for a dashcam footage drive, and allocates the other half to be a music storage drive.
+1. If you are trying to archive on an SFTP/rsync server, then follow these [instructions](SetupRSync.md) and skip step 7. Otherwise, skip this step.
+1. If you are trying to archive on a shared drive, run these commands, subsituting your values for your shared drive:
     ```
     export archiveserver=Nautilus
     export sharename=SailfishCam
     export shareuser=sailfish
     export sharepassword=pa$$w0rd
-    export campercent=100
     ```
 1. If you'd like to receive a text message when your Pi finishes archiving clips follow these [Instructions](ConfigureNotificationsForArchive.md).
 1. Run these commands:

SetupRSync.md

@@ -0,0 +1,43 @@
+# Introduction
+This guide will show you how to utilize [rsync](https://rsync.samba.org/) to archive your saved TeslaCam footage on a remote storage server. In my case, I use this for a networked pi storage server.
+
+This guide makes the following assumptions:
+1. You are running your own ftp/rsync server that you have admin rights to, or can at least add a public key to its `~/.ssh/authorized_keys` file
+1. The ftp/rsync server has rsync installed (raspbian automatically does)
+2. You have **NOT** run the `setup-teslacam` script yet
+# Step 1: Authentication
+Similar to sftp, rsync by default utilizes ssh to connect to a remote server and transfer files. This guide will use a generated ssh keypair, hence the first assumption above.
+
+1. On your teslausb pi, run `ssh-keygen` to generate an ssh key **for the ROOT user!** If you followed the previous steps, you should have already ran `sudo -i` to become the root user on the telsausb pi. If you didn't, run `sudo -i` and re-run `ssh-keygen`. You can be sure that it is generating for root if it asks to store the key in `/root/.ssh/` (versus something like `/home/pi/.ssh`). 
+
+1. Add the contents of the newly generated `/root/.ssh/id_rsa.pub` file from your teslausb pi to the storage server's `~/.ssh/authorized_keys` file. This will allow a nice and easy connection through rsync, no passwords needed!
+
+1. Lastly, you will need to authorize the connection to the FTP/Rsync server and test that the key works, so try connecting to the server (through ssh), and **if it asks if you wish to continue connecting, make sure to type `yes`!** If you do not do this, rsync will fail to connect and thus fail to archive your clips.
+
+# Step 2: Exports
+To be able to configure the teslausb pi use rsync, you'll need to export a few things. On your teslausb pi, run:
+
+```
+export RSYNC_ENABLE=true
+export RSYNC_USER=<ftp username>
+export RSYNC_SERVER=<ftp IP/host>
+export RSYNC_PATH=<destination path to save in>
+```
+
+Explanations for each:  
+* `RSYNC_ENABLE`: `true` for enabling rsync
+* `RSYNC_USER`: The user on the FTP server
+* `RSYNC_SERVER`: The IP address/hostname of the destination machine
+* `RSYNC_PATH`: The path on the destination machine where the files will be saved
+
+An example (of my) config is listed below:
+
+```
+export RSYNC_ENABLE=true
+export RSYNC_USER=pi
+export RSYNC_SERVER=192.168.1.254
+export RSYNC_PATH=/mnt/PIHDD/TeslaCam/
+```
+***Note: RSYNC_ENABLE=true is going to disable the default archive server. Perhaps future releases will allow both to be defined and function at the same time, for redundancy, but for now just pick one that you'll want the most.***
+
+You should be ready to run the setup script now, so return back to step 8 of the [Main Instructions](README.md).

windows_archive/archive-rsync

@@ -0,0 +1,24 @@
+#!/bin/bash -eu
+
+LOG_FILE=/tmp/archive-rsync.log
+
+function log () {
+  echo "$( date )" >> "$LOG_FILE"
+  echo "$1" >> "$LOG_FILE"
+}
+
+source /root/.teslaCamRsyncConfig
+
+nfiles=$(rsync -auvh --stats --log-file=/tmp/archive-rsync-cmd.log /mnt/cam/TeslaCam/saved* $user@$server:$path | awk '/files transferred/{print $NF}')
+
+if [ $nfiles > 0 ]
+then
+    if [ -r "/root/.teslaCamPushoverCredentials" ] && [ $nfiles > 0]
+    then
+        log "Sending Pushover message for copied files."
+        /root/bin/send-pushover $nfiles
+    fi
+    log "Successfully synced files through rsync!"
+else
+    log "No files to archive through rsync!"
+fi

windows_archive/archive-teslacam-clips

@@ -81,11 +81,13 @@ function move_clips_to_archive () {
 
   done
   log "Moved $move_count file(s)."
-  if [ -r "/root/.teslaCamPushoverCredentials" ] && [ $move_count > 0 ]
+
+  if [ -r "/root/.teslaCamPushoverCredentials" ] && [ $move_count > 0]
   then
     log "Sending Pushover message for copied files."
     /root/bin/send-pushover $move_count
   fi
+
   log "Finished moving clips to archive."
 }
 
@@ -121,7 +123,10 @@ function unmount_cam_drive () {
 
 log "Starting..."
 
-ensure_archive_is_mounted
+if [ ! -r "/root/.teslaCamRsyncConfig" ]
+then
+    ensure_archive_is_mounted
+fi
 
 disconnect_usb_drives_from_host
 
@@ -129,7 +134,13 @@ fix_errors_on_cam_drive
 
 ensure_cam_drive_is_mounted
 
-move_clips_to_archive
+if [ -r "/root/.teslaCamRsyncConfig" ]
+then
+    log "Archiving through rsync..."
+    /root/bin/archive-rsync
+else
+    move_clips_to_archive
+fi
 
 unmount_cam_drive
 

windows_archive/setup-teslausb

@@ -105,12 +105,21 @@ function create_usb_drive_backing_files () {
 
 function configure_archive () {
   echo "Configuring the archive..."
+  if [ $RSYNC_ENABLE = true ]
+  then
+    echo "Configuring for Rsync..."
+    echo "user=$RSYNC_USER" > /root/.teslaCamRsyncConfig
+    echo "server=$RSYNC_SERVER" >> /root/.teslaCamRsyncConfig
+    echo "path=$RSYNC_PATH" >> /root/.teslaCamRsyncConfig
+  else
+    echo "Configuring for a shared drive..."
     mkdir /mnt/archive
     local archive_server_ip_address="$(ping -c 1 -w 1 $archiveserver 2>/dev/null | head -n 1 | grep -o -e "(\([[:digit:]]\{1,3\}\.\)\{3\}[[:digit:]]\{1,3\})" | tr -d '()')"
     echo "//$archive_server_ip_address/$sharename /mnt/archive cifs vers=3,credentials=/root/.teslaCamArchiveCredentials,iocharset=utf8,file_mode=0777,dir_mode=0777 0" >> /etc/fstab
 
     echo "username=$shareuser" > /root/.teslaCamArchiveCredentials
     echo "password=$sharepassword" >> /root/.teslaCamArchiveCredentials
+  fi
   echo "Configured the archive."
 }
 
@@ -128,6 +137,12 @@ function configure_archive_scripts () {
   pushd /root/bin
   wget https://raw.githubusercontent.com/"$REPO"/teslausb/"$BRANCH"/windows_archive/archive-teslacam-clips
   chmod +x archive-teslacam-clips
+
+  if [ $RSYNC_ENABLE = true ]
+  then
+    wget https://raw.githubusercontent.com/"$REPO"/teslausb/"$BRANCH"/windows_archive/archive-rsync
+    chmod +x archive-rsync
+  fi 
   popd
   echo "Configured the archive scripts."
 
@@ -190,10 +205,18 @@ function make_root_fs_readonly () {
 
 echo "Verifying environment variables..."
 
+if [ $RSYNC_ENABLE = true ]
+then
+    check_variable "RSYNC_USER"
+    check_variable "RSYNC_SERVER"
+    export archiveserver=$RSYNC_SERVER
+    check_variable "RSYNC_PATH"
+else # Else for now, TODO allow both for more redundancy?
+    check_variable "sharename"
+    check_variable "shareuser"
+    check_variable "sharepassword"
+fi
 check_variable "archiveserver"
-check_variable "sharename"
-check_variable "shareuser"
-check_variable "sharepassword"
 check_variable "campercent"
 
 check_pushover_enabled