From 002d1e6f6cc79ffa83c0b04e0958c774c0496a51 Mon Sep 17 00:00:00 2001 From: Gocnak Date: Sat, 20 Oct 2018 03:26:08 -0400 Subject: [PATCH 1/5] Initial rsync support (untested) Currently untested, but the main idea is there Currently it's either the normal archive server or rsync; not both, which is something to consider when adding things like gdrive and s3 (make it more modular?) --- README.md | 14 +++++++--- SetupRSync.md | 36 +++++++++++++++++++++++++ windows_archive/archive-rsync | 12 +++++++++ windows_archive/archive-teslacam-clips | 17 +++++++++--- windows_archive/setup-teslausb | 37 +++++++++++++++++++++----- 5 files changed, 103 insertions(+), 13 deletions(-) create mode 100644 SetupRSync.md create mode 100644 windows_archive/archive-rsync diff --git a/README.md b/README.md index c672958..9394781 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,10 @@ Set up a share on a Windows (or macOS using Sharing, or Linux using Samba) machi Get the IP address of the archive machine. You'll need this later, so write it down, somewhere. You can do this by opening a command prompt on the archive machine and typing ipconfig. Get the IP address from the line labeled "IPv4 Address". These instructions will assume that the IP address of the archive server is 192.168.0.41. -### TODO Other hosting solutions +### Hosting via SFTP/rsync +Since sftp/rsync is accessing a computer through SSH, the only requirement for hosting an SFTP/rsync server is to have a box running Linux. An example can be another Raspberry Pi connected to your local network with a USB storage drive plugged in. The official Raspberry Pi site has a good example on [how to mount an external drive](https://www.raspberrypi.org/documentation/configuration/external-storage.md). You will need the username and host/IP of the storage server, as well as the path for the files to go in, and the storage server will need to allow SSH. + +### ***TODO: Other hosting solutions*** ## Set up the Raspberry Pi There are four phases to setting up the Pi: @@ -89,13 +92,18 @@ Now that you have Wifi up and running, it's time to set up the USB storage and s ping 192.168.0.41 ``` 1. If you can't ping the archive server by IP address from the Pi, you should go do whatever you need to on your network to fix that. If you can't reach the archive server by name, from the Pi but you can by IP address, then use its IP address, below, in place of its name. -1. Run these commands, subsituting your values. The last line is the percent of the drive you want to allocate for dashcam storage. The remaining percentage will be allocated for music. +1. Determine how much, as a percentage of the drive you want to allocate to recording dashcam footage by using: + ``` + export campercent= + ``` + For example, using `export campercent=100` would allocate 100% of the space to recording footage from your car. `export campercent=50` would be only allocate half of the space for dashcam footage, and allow the other half to be used by music. +1. If you are trying to archive on an SFTP/rsync server, then follow these [instructions](SetupRSync.md), and skip the next step. +1. Run these commands, subsituting your values for your shared drive: ``` export archiveserver=Nautilus export sharename=SailfishCam export shareuser=sailfish export sharepassword=pa$$w0rd - export campercent=100 ``` 1. If you'd like to receive a text message when your Pi finishes archiving clips follow these [Instructions](ConfigureNotificationsForArchive.md). 1. Run these commands: diff --git a/SetupRSync.md b/SetupRSync.md new file mode 100644 index 0000000..9e4888f --- /dev/null +++ b/SetupRSync.md @@ -0,0 +1,36 @@ +# Introduction +This guide will show you how to utilize [rsync](https://rsync.samba.org/) to archive your saved TeslaCam footage on a remote storage server. In my case, I use this for a networked pi storage server. + +This guide makes the following assumptions: +1. You are running your own ftp/rsync server that you have admin rights to, or can at least add a public key to its `~/.ssh/authorized_keys` file +2. You have **NOT** run the `setup-teslacam` script yet +# Step 1: Authentication +Similar to sftp, rsync by default utilizes ssh to connect to a remote server and transfer files. This guide will use a generated ssh keypair, hence the first assumption above. + +On your teslausb pi, run `ssh-keygen` to generate an ssh key. Add the contents of the newly generated `~/.ssh/id_rsa.pub` file from your teslausb pi to the storage server's `~/.ssh/authorized_keys` file. This will allow a nice and easy connection through rsync, no passwords needed! + +# Step 2: Exports +To be able to configure the teslausb pi use rsync, you'll need to export a few things. On your teslausb pi, run: + +``` +export RSYNC_ENABLE=true +export RSYNC_USER= +export RSYNC_SERVER= +export RSYNC_PATH= +``` + +Explanations for each: +* `RSYNC_ENABLE`: `true` for enabling rsync +* `RSYNC_USER`: The user on the FTP server +* `RSYNC_SERVER`: The IP address/hostname of the destination machine +* `RSYNC_PATH`: The path on the destination machine where the files will be saved + +An example (of my) config is listed below: + +``` +export RSYNC_ENABLE=true +export RSYNC_USER=pi +export RSYNC_SERVER=192.168.1.254 +export RSYNC_PATH=/mnt/PIHDD/TeslaCam/ +``` +***Note: RSYNC_ENABLE=true is going to disable the default archive server. Perhaps future releases will allow both to be defined and function at the same time, for redundancy, but for now just pick one that you'll want the most.*** \ No newline at end of file diff --git a/windows_archive/archive-rsync b/windows_archive/archive-rsync new file mode 100644 index 0000000..8881d21 --- /dev/null +++ b/windows_archive/archive-rsync @@ -0,0 +1,12 @@ +#!/bin/bash -eu + +function log () { + echo "$( date )" >> "$LOG_FILE" + echo "$1" >> "$LOG_FILE" +} + +source /root/.teslaCamRsyncConfig + +rsync -au /mnt/archive $user@$server:$path + +log "Successfully synced files through rsync!" \ No newline at end of file diff --git a/windows_archive/archive-teslacam-clips b/windows_archive/archive-teslacam-clips index c6be29a..6383c94 100644 --- a/windows_archive/archive-teslacam-clips +++ b/windows_archive/archive-teslacam-clips @@ -81,11 +81,22 @@ function move_clips_to_archive () { done log "Moved $move_count file(s)." - if [ -r "/root/.teslaCamPushoverCredentials" ] && [ $move_count > 0 ] + + if [ $move_count > 0] then - log "Sending Pushover message for copied files." - /root/bin/send-pushover $move_count + if [ -r "/root/.teslaCamRsyncConfig" ] + then + log "Archiving through rsync..." + /root/bin/archive-rsync + fi + + if [ -r "/root/.teslaCamPushoverCredentials" ] + then + log "Sending Pushover message for copied files." + /root/bin/send-pushover $move_count + fi fi + log "Finished moving clips to archive." } diff --git a/windows_archive/setup-teslausb b/windows_archive/setup-teslausb index f72dc25..1410dca 100644 --- a/windows_archive/setup-teslausb +++ b/windows_archive/setup-teslausb @@ -109,11 +109,20 @@ function create_usb_drive_backing_files () { function configure_archive () { echo "Configuring the archive..." mkdir /mnt/archive - local archive_server_ip_address="$(ping -c 1 -w 1 $archiveserver 2>/dev/null | head -n 1 | grep -o -e "(\([[:digit:]]\{1,3\}\.\)\{3\}[[:digit:]]\{1,3\})" | tr -d '()')" - echo "//$archive_server_ip_address/$sharename /mnt/archive cifs vers=3,credentials=/root/.teslaCamArchiveCredentials,iocharset=utf8,file_mode=0777,dir_mode=0777 0" >> /etc/fstab + if [ $RSYNC_ENABLE = true ] + then + echo "Configuring for Rsync..." + echo "user=$RSYNC_USER" > /root/.teslaCamRsyncConfig + echo "server=$RSYNC_SERVER" > /root/.teslaCamRsyncConfig + echo "path=$RSYNC_PATH" > /root/.teslaCamRsyncConfig + else + echo "Configuring for a shared drive..." + local archive_server_ip_address="$(ping -c 1 -w 1 $archiveserver 2>/dev/null | head -n 1 | grep -o -e "(\([[:digit:]]\{1,3\}\.\)\{3\}[[:digit:]]\{1,3\})" | tr -d '()')" + echo "//$archive_server_ip_address/$sharename /mnt/archive cifs vers=3,credentials=/root/.teslaCamArchiveCredentials,iocharset=utf8,file_mode=0777,dir_mode=0777 0" >> /etc/fstab - echo "username=$shareuser" > /root/.teslaCamArchiveCredentials - echo "password=$sharepassword" >> /root/.teslaCamArchiveCredentials + echo "username=$shareuser" > /root/.teslaCamArchiveCredentials + echo "password=$sharepassword" >> /root/.teslaCamArchiveCredentials + fi echo "Configured the archive." } @@ -131,6 +140,12 @@ function configure_archive_scripts () { pushd /root/bin wget https://raw.githubusercontent.com/"$REPO"/teslausb/"$BRANCH"/windows_archive/archive-teslacam-clips chmod +x archive-teslacam-clips + + if [ $RSYNC_ENABLE = true ] + then + wget https://raw.githubusercontent.com/"$REPO"/teslausb/"$BRANCH"/windows_archive/archive-rsync + chmod +x archive-rsync + fi popd echo "Configured the archive scripts." @@ -193,10 +208,18 @@ function make_root_fs_readonly () { echo "Verifying environment variables..." +if [ $RSYNC_ENABLE = true ] +then + check_variable "RSYNC_USER" + check_variable "RSYNC_SERVER" + archiveserver = $RSYNC_SERVER + check_variable "RSYNC_PATH" +else # Else for now, TODO allow both for more redundancy? + check_variable "sharename" + check_variable "shareuser" + check_variable "sharepassword" +fi check_variable "archiveserver" -check_variable "sharename" -check_variable "shareuser" -check_variable "sharepassword" check_variable "campercent" check_pushover_enabled From b60b78795339d122a1ea65276c514424b795882c Mon Sep 17 00:00:00 2001 From: Gocnak Date: Sat, 20 Oct 2018 04:27:44 -0400 Subject: [PATCH 2/5] Minor fixes Archive isn't mounted for rsync, it's just a folder --- windows_archive/archive-teslacam-clips | 7 +++++-- windows_archive/setup-teslausb | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/windows_archive/archive-teslacam-clips b/windows_archive/archive-teslacam-clips index 6383c94..a9c6b4d 100644 --- a/windows_archive/archive-teslacam-clips +++ b/windows_archive/archive-teslacam-clips @@ -96,7 +96,7 @@ function move_clips_to_archive () { /root/bin/send-pushover $move_count fi fi - + log "Finished moving clips to archive." } @@ -132,7 +132,10 @@ function unmount_cam_drive () { log "Starting..." -ensure_archive_is_mounted +if [ ! -r "/root/.teslaCamRsyncConfig" ] +then + ensure_archive_is_mounted +fi disconnect_usb_drives_from_host diff --git a/windows_archive/setup-teslausb b/windows_archive/setup-teslausb index 1410dca..428b3cf 100644 --- a/windows_archive/setup-teslausb +++ b/windows_archive/setup-teslausb @@ -212,7 +212,7 @@ if [ $RSYNC_ENABLE = true ] then check_variable "RSYNC_USER" check_variable "RSYNC_SERVER" - archiveserver = $RSYNC_SERVER + export archiveserver=$RSYNC_SERVER check_variable "RSYNC_PATH" else # Else for now, TODO allow both for more redundancy? check_variable "sharename" From de3371f00caa060780fcee8d82114b3de0fa8212 Mon Sep 17 00:00:00 2001 From: Gocnak Date: Sat, 20 Oct 2018 05:30:57 -0400 Subject: [PATCH 3/5] rsync will not be using the mnt/archive folder It'll instead bypass and just call itself on its own --- windows_archive/archive-rsync | 14 ++++++++++++-- windows_archive/archive-teslacam-clips | 23 ++++++++++------------- windows_archive/setup-teslausb | 2 +- 3 files changed, 23 insertions(+), 16 deletions(-) diff --git a/windows_archive/archive-rsync b/windows_archive/archive-rsync index 8881d21..b076721 100644 --- a/windows_archive/archive-rsync +++ b/windows_archive/archive-rsync @@ -7,6 +7,16 @@ function log () { source /root/.teslaCamRsyncConfig -rsync -au /mnt/archive $user@$server:$path +nfiles=$(rsync -auh --stats /mnt/cam/TeslaCam/saved* $user@$server:$path | awk '/files transferred/{print $NF}') -log "Successfully synced files through rsync!" \ No newline at end of file +if [ $nfiles > 0 ] +then + if [ -r "/root/.teslaCamPushoverCredentials" ] && [ $nfiles > 0] + then + log "Sending Pushover message for copied files." + /root/bin/send-pushover $nfiles + fi + log "Successfully synced files through rsync!" +else + log "No files to archive through rsync!" +fi \ No newline at end of file diff --git a/windows_archive/archive-teslacam-clips b/windows_archive/archive-teslacam-clips index a9c6b4d..23bdf4d 100644 --- a/windows_archive/archive-teslacam-clips +++ b/windows_archive/archive-teslacam-clips @@ -82,19 +82,10 @@ function move_clips_to_archive () { done log "Moved $move_count file(s)." - if [ $move_count > 0] + if [ -r "/root/.teslaCamPushoverCredentials" ] && [ $move_count > 0] then - if [ -r "/root/.teslaCamRsyncConfig" ] - then - log "Archiving through rsync..." - /root/bin/archive-rsync - fi - - if [ -r "/root/.teslaCamPushoverCredentials" ] - then - log "Sending Pushover message for copied files." - /root/bin/send-pushover $move_count - fi + log "Sending Pushover message for copied files." + /root/bin/send-pushover $move_count fi log "Finished moving clips to archive." @@ -143,7 +134,13 @@ fix_errors_on_cam_drive ensure_cam_drive_is_mounted -move_clips_to_archive +if [ -r "/root/.teslaCamRsyncConfig" ] +then + log "Archiving through rsync..." + /root/bin/archive-rsync +else + move_clips_to_archive +fi unmount_cam_drive diff --git a/windows_archive/setup-teslausb b/windows_archive/setup-teslausb index 428b3cf..4fa2540 100644 --- a/windows_archive/setup-teslausb +++ b/windows_archive/setup-teslausb @@ -108,7 +108,6 @@ function create_usb_drive_backing_files () { function configure_archive () { echo "Configuring the archive..." - mkdir /mnt/archive if [ $RSYNC_ENABLE = true ] then echo "Configuring for Rsync..." @@ -117,6 +116,7 @@ function configure_archive () { echo "path=$RSYNC_PATH" > /root/.teslaCamRsyncConfig else echo "Configuring for a shared drive..." + mkdir /mnt/archive local archive_server_ip_address="$(ping -c 1 -w 1 $archiveserver 2>/dev/null | head -n 1 | grep -o -e "(\([[:digit:]]\{1,3\}\.\)\{3\}[[:digit:]]\{1,3\})" | tr -d '()')" echo "//$archive_server_ip_address/$sharename /mnt/archive cifs vers=3,credentials=/root/.teslaCamArchiveCredentials,iocharset=utf8,file_mode=0777,dir_mode=0777 0" >> /etc/fstab From 1648a1993fa2d83ac9602a38b48476f75069c3e9 Mon Sep 17 00:00:00 2001 From: Gocnak Date: Sat, 20 Oct 2018 06:45:08 -0400 Subject: [PATCH 4/5] More minor fixes Fixed rsync user and server being overwritten Added log file outputs for archive-rsync --- windows_archive/archive-rsync | 4 +++- windows_archive/setup-teslausb | 4 ++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/windows_archive/archive-rsync b/windows_archive/archive-rsync index b076721..d163640 100644 --- a/windows_archive/archive-rsync +++ b/windows_archive/archive-rsync @@ -1,5 +1,7 @@ #!/bin/bash -eu +LOG_FILE=/tmp/archive-rsync.log + function log () { echo "$( date )" >> "$LOG_FILE" echo "$1" >> "$LOG_FILE" @@ -7,7 +9,7 @@ function log () { source /root/.teslaCamRsyncConfig -nfiles=$(rsync -auh --stats /mnt/cam/TeslaCam/saved* $user@$server:$path | awk '/files transferred/{print $NF}') +nfiles=$(rsync -auvh --stats --log-file=/tmp/archive-rsync-cmd.log /mnt/cam/TeslaCam/saved* $user@$server:$path | awk '/files transferred/{print $NF}') if [ $nfiles > 0 ] then diff --git a/windows_archive/setup-teslausb b/windows_archive/setup-teslausb index 4fa2540..829f631 100644 --- a/windows_archive/setup-teslausb +++ b/windows_archive/setup-teslausb @@ -112,8 +112,8 @@ function configure_archive () { then echo "Configuring for Rsync..." echo "user=$RSYNC_USER" > /root/.teslaCamRsyncConfig - echo "server=$RSYNC_SERVER" > /root/.teslaCamRsyncConfig - echo "path=$RSYNC_PATH" > /root/.teslaCamRsyncConfig + echo "server=$RSYNC_SERVER" >> /root/.teslaCamRsyncConfig + echo "path=$RSYNC_PATH" >> /root/.teslaCamRsyncConfig else echo "Configuring for a shared drive..." mkdir /mnt/archive From f335f236115586ea4ba06cd173721d6614442ffb Mon Sep 17 00:00:00 2001 From: Gocnak Date: Sat, 20 Oct 2018 22:58:30 -0400 Subject: [PATCH 5/5] Updated instructions to prevent bizarre deadend Need to make sure you use the root user to generate the ssh key, and make sure to authorize the connection initially --- README.md | 8 ++++---- SetupRSync.md | 11 +++++++++-- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 9394781..220645a 100644 --- a/README.md +++ b/README.md @@ -92,13 +92,13 @@ Now that you have Wifi up and running, it's time to set up the USB storage and s ping 192.168.0.41 ``` 1. If you can't ping the archive server by IP address from the Pi, you should go do whatever you need to on your network to fix that. If you can't reach the archive server by name, from the Pi but you can by IP address, then use its IP address, below, in place of its name. -1. Determine how much, as a percentage of the drive you want to allocate to recording dashcam footage by using: +1. Determine how much, as a percentage, of the drive you want to allocate to recording dashcam footage by using: ``` export campercent= ``` - For example, using `export campercent=100` would allocate 100% of the space to recording footage from your car. `export campercent=50` would be only allocate half of the space for dashcam footage, and allow the other half to be used by music. -1. If you are trying to archive on an SFTP/rsync server, then follow these [instructions](SetupRSync.md), and skip the next step. -1. Run these commands, subsituting your values for your shared drive: + For example, using `export campercent=100` would allocate 100% of the space to recording footage from your car, and would not create a separate music partition. `export campercent=50` would be only allocate half of the space for a dashcam footage drive, and allocates the other half to be a music storage drive. +1. If you are trying to archive on an SFTP/rsync server, then follow these [instructions](SetupRSync.md) and skip step 7. Otherwise, skip this step. +1. If you are trying to archive on a shared drive, run these commands, subsituting your values for your shared drive: ``` export archiveserver=Nautilus export sharename=SailfishCam diff --git a/SetupRSync.md b/SetupRSync.md index 9e4888f..7106a7a 100644 --- a/SetupRSync.md +++ b/SetupRSync.md @@ -3,11 +3,16 @@ This guide will show you how to utilize [rsync](https://rsync.samba.org/) to arc This guide makes the following assumptions: 1. You are running your own ftp/rsync server that you have admin rights to, or can at least add a public key to its `~/.ssh/authorized_keys` file +1. The ftp/rsync server has rsync installed (raspbian automatically does) 2. You have **NOT** run the `setup-teslacam` script yet # Step 1: Authentication Similar to sftp, rsync by default utilizes ssh to connect to a remote server and transfer files. This guide will use a generated ssh keypair, hence the first assumption above. -On your teslausb pi, run `ssh-keygen` to generate an ssh key. Add the contents of the newly generated `~/.ssh/id_rsa.pub` file from your teslausb pi to the storage server's `~/.ssh/authorized_keys` file. This will allow a nice and easy connection through rsync, no passwords needed! +1. On your teslausb pi, run `ssh-keygen` to generate an ssh key **for the ROOT user!** If you followed the previous steps, you should have already ran `sudo -i` to become the root user on the telsausb pi. If you didn't, run `sudo -i` and re-run `ssh-keygen`. You can be sure that it is generating for root if it asks to store the key in `/root/.ssh/` (versus something like `/home/pi/.ssh`). + +1. Add the contents of the newly generated `/root/.ssh/id_rsa.pub` file from your teslausb pi to the storage server's `~/.ssh/authorized_keys` file. This will allow a nice and easy connection through rsync, no passwords needed! + +1. Lastly, you will need to authorize the connection to the FTP/Rsync server and test that the key works, so try connecting to the server (through ssh), and **if it asks if you wish to continue connecting, make sure to type `yes`!** If you do not do this, rsync will fail to connect and thus fail to archive your clips. # Step 2: Exports To be able to configure the teslausb pi use rsync, you'll need to export a few things. On your teslausb pi, run: @@ -33,4 +38,6 @@ export RSYNC_USER=pi export RSYNC_SERVER=192.168.1.254 export RSYNC_PATH=/mnt/PIHDD/TeslaCam/ ``` -***Note: RSYNC_ENABLE=true is going to disable the default archive server. Perhaps future releases will allow both to be defined and function at the same time, for redundancy, but for now just pick one that you'll want the most.*** \ No newline at end of file +***Note: RSYNC_ENABLE=true is going to disable the default archive server. Perhaps future releases will allow both to be defined and function at the same time, for redundancy, but for now just pick one that you'll want the most.*** + +You should be ready to run the setup script now, so return back to step 8 of the [Main Instructions](README.md). \ No newline at end of file